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ABSTRACT 

At the request of the Office of the Assistant 
Secretary for Educational Research and Improvement , the Panel to 
Evaluate the National Center for Education Statistics (NCES) was 
established in 1985 by the National Academy of Sciences, This report 
culminates an extensive two-year study of NCES and reflects the 
panel's broad view of the center's serious problems. The study had 
four major aspects: (1) to review, describe, and assess existing data 
quality and quality assurance processes; (2) to review, describe, and 
assess program content and services and procedures used for 
establishing priorities consistent with the Congressional mandate; 
(3) to review and assess the timeliness of data collected and 
disseminated; and (4) to identify issues hindering NCES in the 
successful accomplishment of its mission. The panel recognized that 
the center will require increased funding, bat funding requests 
should be judged against the progress of the center in meeting 
standards; and reflect the expected expansion of data collection. The 
panel's comprehensive recommendations addressed the following areas: 
(1) mission, role, and responsibilities; (2) content and direction of 
data collection programs; (3) establishing a frame of reference; (4) 
the advisory council; (5) improving data quality (redesigning the 
elementary /secondary data programs, reducing error, and statistical 
standards); (6) timeliness; (7) staff and bmdget resources; (8) 
contracting out; and (9) publications policy* The panel i ecommt nded 
that serious consideration be given to abolishing the center and 
finding other means to obtain and disseminate education data if 
wide-ranainq actions are not undertaken to change both the image and 
reality of the center. Appendices include the center's organizational 
structure and biographical sketches of panel and staff. (XMD) 
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Summary Statement of the Panel 



In late 1984 the National Academy of Sciences was asked by 
the Office of the Assistant Secretary for Educational 
Research and Improvement of the U.S. Department of 
Education to undertake an evaluation of the National Center 
for Education Statistics (NCES). The request was a 
reflection of the expressed concern that the center had lost 
the confidence both of those providing data to it and of 
those who used its products, that the quality of its products 
had declined, that it failed to provide its data in a timely 
fashion, and that its interpretations of those data it did pro- 
vide were flawed. 

At the same time, an educational reform movement had 
brought education statistics into what is likely to be the 
center of policy debate for years to come. The educational 
reform movement burst onto the public agenda in early 1983 
with the issuance of the first of the reform reports from the 
Department of Education's Commission on Excellence in Edu- 
cation (U.S. Department of Education, 1983). The spate of 
reports that followed from a multitude of groups laying claim 
to the issue relied on data from the center, among other 
sources, to provide the evidence for declines in quality and 
to argue for specific solutions, such as increasing high school 
graduation requirements in mathematics and science. 

This situation provides a dramatic illustration of the 
power of data tb fuel a policy debate and of the changing 
demands for data that come with the recognition of a crisis. 
How can policy makers identify needed intervention points? 
How can society increase the amount of educational output? 
How will people know if reforms have been effective? The 
demand for data to help answer these and other questions 
has thrust the center forward and promises to keep it there 
for a long time. 
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To carry out the study requested by the department, the 
Panel to Evaluate the National Center for Education Statis- 
tics was established in January 1985 under the Committee on 
National Statistics of the National Research Council. The 
study had four major aspects: 

o to review, describe, and assess existing data quality 
and quality assurance processes by NCES, including 
standards and guidelines for conducting surveys and 
releasing and publishing the data, and procedures 
used to ensure accuracy, reliability, and validity of 
the data as well as ensuring consistency and 
uniformity over time. 

o to review, describe, and assess program content and 
services of NCES and the procedures used for estab- 
lishing priorities consistent with the congressional 
mandate. 

o to review and assess the timeliness of data collected 
and disseminated by NCES. 

o to identify any issues that obstruct or hinder NCES 
in the successful accomplishment of its mission. 

The panel's activities included interviews with appropriate 
officials and experts within the center, the Department of 
Education, and the federal government; with members of the 
department's Advisory Council on Education; with representa- 
tives of many of the education organizations and professional 
groups concerned with education; with other users of educa- 
tion data; and with providers of the data to the center. The 
panel drew on the expertise and advice of other committees 
and panels of the National Research Council whose interests 
extend to the use of education statistics. The panel also 
reviewed an extensive body of materials, including the annual 
reports of the advisory council, reports of previous reviews, 
internal memoranda, and relevant correspondence. 

The panel began its activities with the full intention of 
approaching the issues through examination of individual pro- 
grams. We soon found, however, that the center's problems, 
rather than being unique to particular programs, extended 
throughout the center. Thus our report focuses on issues 
rather than programs. 

The report culminates an extensive two-year study and 
reflects the panel's very broad view of the problems of the 
center. We have addressed the problems both from the reali- 
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ties of the past and present and from the longer-range per- 
spective of the center as we perceive it in the years ahead. 
Simply put, we have found the center wanting. 

We believe that the discussion and recommendations in our 
report provide a realistic approach to dealing with the issues 
and problems that the center faces and that actions such as 
those we recommend are critical to its survival as a credible 
organization. Neither our findings nor many of our recom- 
mendations are startling in nature; rather, they tend to echo 
or expand what has been said by previous groups that have 
explored the same issues. However, because we believe edu- 
cation issues are now so urgent, we place more emphasis on 
comprehensive action than did past reviews. 

Given that the need for information about education is 
beyond argument, in this report \*e outline a goal that is 
within the grasp of a statistical agency working in the dif- 
ficult environment of the federal government with its dispar- 
ate objectives and multitudinous demands on budgets and 
staff. To a large extent, the actions we propose, to a large 
extent, are inherent elements in the operating philosophies 
that guide such respected statistical organizations as the 
Bureau of the Census, the Bureau of Labor Statistics, the 
National Center for Health Statistics, and the Energy Inform- 
ation Administration. In essence, we seek to institute 
changes at the Center for Statistics for which there is ample 
precedent. 

As we make clear in the report, the center's problems 
extend throughout its full program; thus, our recommenda- 
tions are a comprehensive and integrated set. Our proposals 
provide a package that, when taken as a whole, will initiate 
and reinforce a process of improvement. The approach is 
realistic-not radical. The changes we propose cannot be 
effected overnight, or even in a year. But they can begin 
with the initial and fundamental steps of improvement 

Simple tinkering— selecting a suggestion here, reorganizing 
a bit there, adding a memo, deleting a regulation-has not 
worked and will not work, as the history of the past 25 
years shows. Such self-deception, represented by quick-fix 
approaches, merely reinforces the already unacceptable status 
quo. The overriding issue before the Department of Educa- 
tion and the Center for Statistics is one of intent, commit- 
ment, and leadership. 

It is appropriate at this point to state unequivocally that 
we are not proposing to solve the center's problems merely 
by throwing money at them. While the center does require 
immediate additional staff, and thus some incremental addi- 
tional funding, just to carry out its current workload, we 
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firmly believe that the center, by implementing many of our 
recommendations, can significantly improve its operations 
without additional large infusions. Certainly, the panel 
recognizes and accepts that over time the center will require 
increased funding, but such requests, first, should be judged 
against the progress of the center in meeting the standards 
expected of it and, second, should reflect an expected expan- 
sion in the types and scope of data the center collects as it 
demonstrates an ability to meet the diverse needs of its 
users. 

We wish to emphasize the seriousness with which we view 
the center's problems. We believe that there can be no 
defense for allowing the center to continue as it has for all 
too long. If, indeed, "the nation is at risk" in the area of 
education, it is past time for those in positions of responsi- 
bility to acknowledge the risks and dangers of perpetuating 
the myriad and continuing problems of the center. Without 
strong and continuous commitment and demonstrated determ- 
ination to undertake wide-ranging actions to change both the 
image and reality of the center, we are unanimous in our 
conviction that serious consideration should be given to the 
more drastic alternatives of abolishing the center and finding 
other means to obtain and disseminate education data. A 
number of potential alternatives to the center exist, such as 
placing leadership responsibility elsewhere in the department 
and collecting the needed data through contracting with 
another federal statistical agency or with the private sector, 
through a federally funded nonprofit research center 
established by the department at the direction of the Con- 
gress, or through directing other entities within the depart- 
ment to accomplish the task. 

We emphasize strongly, however, that we believe the pre- 
ferred course of action is to begin the process of improve- 
ment. As we have noted, the center's problems are long- 
standing and pervasive, but if faced openly they can, in time, 
be overcome. As for this report-and its many predecessors 
-perhaps our thought is best expressed in the words of Sir 
Francis Bacon: 

Crafty men condemn studies 
Simple men admire them 
Wise men use them. 
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Education Statistics: 
History and Problems 



A BRIEF HISTORY 



In 1S67 Congress established a Department of Education and, 
in the first section of the legislation, gave it a primary mis- 
sion of "collecting such statistics and facts as shall show the 
condition and progress of education in the several States and 
Territories, and of diffusing such information respecting the 
organization and management of schools and school systems 
and methods of teaching." Congress further provided for the 
Commissioner of Education to present annually "a report 
embodying the results of his investigations and labors, 
together with a statement of such facts and recommendations 
as will, in his judgment, subserve the purpose for which this 
department is established.** Thus in the department's earliest 
incarnation, the need for and importance of statistics was 
both recognised and established. 

During the ensuing 30 years, the Commissioners of Educa- 
tion focused the efforts of what by then had become a 
Bureau of Education on the collection and reporting of school 
statistics, such that one scholar observed "that the phrase 
[reporting of school statistics] became a definition of the 
bureau itself** (Warren, 1974:163). The first index to its pub- 
lications was issued in 1891 and covered the Bureau's efforts 
between 1867 and 1890. Containing over 100 pages, the list- 
ings are almost equally divided among historical sketches, 
descriptive pieces, and statistical summaries (U.S. Bureau of 
Education, 1891). Even at this early point there was recog- 
nition of the need to interpret the facts. For example, in 
its annual report for i897, two chapters were devoted to a 
descriptive analysis of program data acquired by the office; 
in 1902, the annual report dealt with the subject of black 
student enrollment in higher education; and in 1903, the edu- 
cation of women was a focus of discussion. 
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This focus on the collection and dissemination of educa- 
tion statistics by the Bureau of Education (and, later, the 
Office of Education) continued more or less unabated 
throughout the first half of the twentieth century, even as 
the organization lost its independent status and became a 
part of the Department of the Interior (1868) and eventually 
settled within the Federal Security Agency (1939), which 
later became the Department of Health, Education, and 
Welfare. 

The National Center for Education Statistics appeared in 
January 1965, having been established as a staff office 
reporting to the Commissioner of Education. The idea for a 
center originated from the recommendations of a public advi- 
sory commission and a White House task force "to provide 
for education, the statistical expertise and products which 
the Bureau of Labor Statistics provides for Labor, the 
National Center for Health Statistics for Health, and the 
Agricultural Reporting Service for Agriculture" (Ottina, 1973). 
The center assumed responsibility for all of the general pur- 
pose data gathering within the Office of Education and for 
helping states, through a program of grants, improve their 
own data-gathering and statistical capabilities. 

The federal education programs of the Great Society, ini- 
tiated during the administration of President Johnson, placed 
new and heavy demands on the very limited resources of the 
young center. However, the needs of the center for addi- 
tional staff and funding failed to win the support of the 
Congress. Rather, Congress sought to strengthen the center 
by changing it in 1974 to a statutory entity reporting 
directly to the Assistant Secretary for Education, With the 
establishment of a separate Department of Education in early 
1980, the center became part of Office of Educational 
Research and Improvement and continued to report to an 
assistant secretary. In October 1985 yet another reorganiza- 
tion was carried out which, among other changes, resulted in 
the renaming of the National Center for Education Statis- 
tics— "henceforth it will be known as the 'Center for Statis- 
tics*"— and removed some of its information dissemination 
functions. As is always the case, some internal restructuring 
occurred, with appropriate name changes for the operating 
elements. A description of the organizational structure of 
the center, as of April 1986, is in Appendix A. 
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WHAT THE CENTER COLLECTS 



Until the late 1960s the statistical programs of the center 
and its predecessors were restricted primarily to general 
purpose data— enrollments, staff, and finance in the elemen- 
tary, secondary, and postsecondary schools. The past quarter 
century, however, has seen the demands for additional and 
different kinds of information, which has resulted both in the 
initiation of special scudies designed to meet the specific 
needs of Congress and in the development of new and expan- 
ded data collection systems at all levels of education. For 
example, the elementary/secondary school program was 
enlarged and expanded to reflect the effects of preprimary 
education and of adult and noncollegiate postsecondary edu- 
cation, to provide data on libraries and, most importantly, to 
implement longitudinal studies to measure and report on the 
effects of cumulative student development and change. In 
addition, information on inputs (such as enrollments, number 
of schools, and funding) and process (such as expenditures 
and curricula) was supplemented to reflect the increased 
emphasis in outcome measures—what students learn, what 
they knov, and what happens to them after they complete 
their formal education and training. During this period, the 
center also initiated a series of publications, established a 
fast response survey system to meet data needs for policy 
determination, became responsible for the National Assess- 
ment of Educational Progress (NAEP) and, overall, grew in 
program and importance. The center lost responsibility for 
NAEP 'ater in the decade, but regained it in October 1985. 
The principal center programs at the time the panel's study 
was initiated are described briefly below. 



The Common Core of Data (CCD) program currently is the 
primary source of basic statistical data about public elemen- 
tary and secondary educational institutions. Much of the 
data is derived from administrative records maintained by the 
state education agencies (SEAs), which compile the data in 
the desired formats and transmit them to the center. The 
CCD consists of the following six separate parts. 

Part I. The Public School Universe File . This annual 
census of SEAs provides information on all public elementary 
and secondary schools in operation during a school year. 
This file includes information on school type, grade span, fall 
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membership, and the number of classroom teachers. The 
school data are aggregated to the state level. 

Part II. Local Education Agencv (LEA) Universe File . 
This annual census of SEAs provides information for the uni- 
verse of LEAs in a state on :vpe of agency, operating status, 
fiscal status, and control sr*. The LEA data are aggre- 
gated to the state level. 

Part III. Local Education Agency Nonfiscal Report . This 
annual census of SEAs provides data on LEAs on number of 
schools operated by the agency, membership, and full-time 
equivalent teachers and other staff. The LEA data are 
aggregated to the state level. 

Part IV. Public School District Finance Report . This 
annual census of SEAs provides data on LEAs on average 
daily attendance, revenues by source, and expenditures by 
major function. 

Part V. State Aggregate Nonfiscal Report . This annual 
census of SEAs provides data for states on fall membership 
by grade level, full-time-equivalent staff by major category, 
and high school graduates. 

Part VI. State Aggregate Fiscal Report . This annual 
census of SEAs and other state agencies that provide 
resources to LEAs provides aggregate data for states on 
average daily attendance, school district revenues by source, 
and expenditures by major function. 



Sample Surveys 

In addition to the administrative data collected through the 
Common Core of Data, the center conducts sample surveys 
among education institutions to obtain other data on public 
and private elementary and secondary education to address 
emerging policy issues. Some of these surveys are done 
periodicall.*: others are of a one-time nature. Some recent 
examples include: 

Public School Survey. This survey of a nationally 
representative sample of both schools and teachers within the 
schools obtained core summary data and supplementary data 
on topics of policy interest. The core data collected from 
school administrators included: grade span, fall membership, 
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minority enrollment, full-time equivalent (FTE) teachers and 
other school staff, class size, and high school graduates- 
Data collected from teachers included: age, sex, and race; 
education and training; teaching assignments; time utilization; 
and compensation. The data from the surveys of schools and 
teachers were aggregated to the national level. 

Private School Survey . This survey of a nationally 
representative sample of private elementary and secondary 
schools, supplemented by an area sample of schools, obtained 
core school summary data and supplementary data on topics 
of policy interest. The data collected on schools were, in 
general, parallel to data collected in the public school survey, 
but the survey also included data on scnool policies, program 
offerings, tuition rates, and participation in federal program. 
As in the public school survey, data were aggregated to the 
national level. 

Survey of Teacher Demand and Shortage . This survey of 
a nationally representative sample of LEAs and private 
schools obtained data on the number of teachers by 
assignment, the number of new hires, the number of positions 
that ould not be fiir.d, and recruiting and employment 
practices. The data from LEAs and private schools were 
aggregated to form national estimates. 



National Assessment of Education Progress 

The National Assessment of Educational Progress (NAEP) was 
mandated by Congress as a program to continuously monitor 
the knowledge, skills, and performance of the nation's 
children and youth. Begun in 1969, NAEP has each year 
conducted assessments of students at ages 9, 13, and 17 in 
ten school-related subject areas: reading, mathematics, writ- 
ing, science, citizenship, social studies, art, music, literature, 
and career and occupational development. NAEP also 
conducts special assessments in other educational areas, such 
as young adult literacy, which is an assessment of the nature 
and extent of literacy among young adults aged 21-25. 



Longitudinal Surveys 

Beginning in 1972 the center initiated the first in a series of 
longitudinal studies, known as the National Longitudinal 
Study of the High School Class of 1972. The fifth follow-up 
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survey of this study was conducted in 1985. The second 
study, High School and Beyond, is a national longitudinal 
study of cohorts of 1980 high school sophomores and seniors, 
who are being interviewed over time to determine what 
happened to them after high school completion. Parents, 
teachers, and school administrators were also interviewed in 
connection with the study. Beginning in 1988, this survey 
will be complemented by the National Education Longitudinal 
Survey (NELS:88), which will begin with an eighth grade 
cohort. The major objectives of the longitudinal studies is 
to provide trend info; .nation about the quality, equality, and 
diversity of educational opportunity and the effect of these 
factors on individual development and educational and career 
outcomes; changes over time in educational and career 
outcomes; and transition rates for such changes as high 
school to college, two-year colleges to four-year colleges and 
universities, and school to work. 



Higher Education G;neral Information Survey 

The Higher Education General Information Survey (HEGIS), 
begun in the mid-1960s, is a coordinated effort designed to 
acquire and maintain statistical data on the characteristics 
and operations of institutions of higher education. A series of 
surveys, HEGIS collects information f rom the universe of 
institutions of higher education on five subjects: institu- 
tional characteristics of colleges and universities; opening fall 
enrollment; salaries, tenure, and fringe benefits of full-time 
instructional faculty; degrees and other formal awards con- 
ferred; and financial statistics. 

Concurrent with the conduct of HEGIS, the center was 
developing a new program, titled the Integrated Postsecon- 
dary Education Data System (IPEDS), which will replace 
HEGIS and provide a data base covering Doth traditional and 
nontraditional postsecondary institutions. A major rationale 
for the development of an integrated system is its improve- 
ment over some currently used surveys of postsecondary edu- 
cation that are recognized as having statistical deficiencies. 
As the panel's study was under way, forms and procedures 
had been established and a pilot study successfully completed. 
Plans call for this revised approach to be introduced to the 
postsecondary education community beginning in late 1986. 
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Vocational Education Data System 



Mandated in 1976, the objective of the Vocational Education 
Data System (VEDS) program was to collect data on voca- 
tional education for purposes of policy analysis, program 
accountability, program evaluation, and manpower planning. 
The program did not meet the expectations, despite under- 
going a number of revisions, and was terminated in December 
1983. At the present time, the center is exploring a number 
of alternative means of collecting data on vocational educa- 
tion. 



In addition to the sources described above, the center 
conducts ad hoc surveys, as needed, on subjects of immediate 
interest or obtains data under contractual agreements with 
SEAs, with private contractors, or with one of the federal 
statistical agencies. 

It also must be borne in mind that the data collected by 
the center represents only a relatively small proportion of 
the data collected by the department, since progiam offices 
directly collect program and compliance data. 

With the appointment of a new administrator in late 1984, 
the center initiated & thorough review ui all of its data col- 
lection programs, addressing issues of suitability, scope, relia- 
bility, and timeliness of its products from the viewpoints of 
the variety of purposes to be served, including instructional, 
administrative, and policy. This endeavor is unique in that it 
incorporates direct public comment and participation by the 
education community (and by other interested parties) on all 
aspects of the center's programs, from objectives through 
data needs and from collection methodologies to output and 
product. The extent to which this approach is successful and 
is reflected in the future data-gathering and information-pro- 
ducing activities of the center cannot be assessed at this 
time, but the panel commends the center leadership for 
taking this step and for attempting to involve its community 
of interested and concerned users in the development of the 
model which will guide its program for the future. 



THE CENTER--WHOM SHOULD IT SERVE? 

With 120 years of tradition to call upon, the appropriateness 
of the federal government's role in the collection and provi- 
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sion oT statistical information on education is no longer an 
issue. Recently, in fact, a report by the Heritage Foundation 
calling for the abolition of the Department of Education 
categorically exempted the collection of statistics (Butler et 
al., 1984). Nonetheless, there has been, and continues to be, 
ongoing and ofttimes strident debate, if not confusion, on the 
more fundamental issue, namely, the role of a center for 
education statistics: What is it? What should it be? 

The arguments and disagreements are long standing and, 
iike demons, appear and reappear in a variety of forms. 
Perhaps the broadest and most fundamental disagreement is 
expressed as the conflict between two philosophies: one that 
maintains the availability of wide and ready access to basic 
information is an important federal responsibility, and one 
that justifies federal data collection only as required to meet 
specific program needs. That this is not a new' problem is 
borne out by reference to a management evaluation conducted 
by the Department of Health, Education and Welfare in 1978, 
which noted "confused opinions on what NCES' role is and 
should be in relation to the Education Division, the Office of 
the Secretary, and Congressional mandate . . ." The report 
went on to note: "this confusion ... is a result of one 
major factor: the lack of an established role in the Depart- 
ment for NCES" (U.S. Department of Health, Education, and 
Welfare, 1978:13). This concern also was expressed by the 
Advisory Council on Education Statistics (ACES) in its second 
annual report, issued in 1977. The council expressed its 
concern in terms of the degree of autonomy and indepen- 
dence available to NCES, to "insure its products and activi- 
ties have an actual and perceived non-political character" and 
stated, further, that its role in "collecting data for purposes 
of compliance enforcement, program evaluation or regulation 
. . . must not impair the effectiveness of the center as a 
gene r al purpose statistical agency" (U.S. Department of 
Health, Education, and Welfare, 1977:6). 

A clear view of the role of a federal statistical agency, in 
the panel's view, is found in remarks presented at the Census 
Bureau's first annual research conference in March 1985. In 
discussing the role of statistics in government decision 
making, the then Under Secretary for Economic Affairs of 
the Department of Commerce, Dr. Sidney L. Jones (1985), 
said: 

I believe that the mission of statistical agencies should 
be to fulfill their specific responsibilities within the 
overall information system, rather than responding to 
the ad hoc functional priorities of their parent institu- 
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tions. . . . These two institutions (Census and the 
Bureau of Economic Analysis), along with others like 
the Bureau of Labor Statistics, the Bureau of Justice 
Statistics, the National Centers of Health and Educa- 
tion Statistics, and the Statistical Reporting Service of 
the Department of Agriculture, are all part of an 
information system whose principal responsibility is the 
integrity of the numbers, not responsiveness to politi- 
cal needs. That should be their number-one priority. 
The fact that they happen to be parts of . . . some 
[other] department is, to me, not the controlling fact 
of their missions. 

The panel fully shares Dr. Jones' view. To restrict the 
center solely or mainly to the data requirements of those 
with program responsibility, in the panel's opinion, would be 
extremely shortsighted. It would result in a statistical 
organization stilted in scope, limited in flexibility, and viewed 
as a partisan arm of the department. And it v*ould violate 
the history of and congressional intent for the center. 



THE CENTER'S PROBLEMS 

The center has experienced relatively little success over the 
yeirs in obtaining support for funds, staff, or program initia- 
tives from its department, from the Office of Management 
and Budget, or from Congress. This lack of support has con- 
tributed to the inability of the center to develop both the 
image and the reality of a competent and objective major 
statistical organization serving the wide need for statistics 
about education in the United States. The reason for persis- 
tent lack of support is not clear. One suggestion is that the 
highly decentralized nature of education has placed the gath- 
ering of statistics at the vortex of a continuing struggle, a 
tug-of-war, if you will, between the diverse players in the 
education field— local and state officials, associations and 
interest groups, the Congress, and groups with program 
responsibilities within the department—few of whom could or 
did agree on what should or could be collected, or how. A 
recent report by the Committee for Economic Development, 
Investing it; Our Children, summarized the situation in these 
words: "Private industry could not succeed with a data-col- 
lection system and research base as weak as this nation has 
in the field of education" (quoted in Anthropological 
Newsletter 26(9), 1985). 



23 



14 



Reports of prior advisory committees--in 1957, in 1960, 
and in 1 963-3 nd the report of an internal departmental 
study in 1978 all testify to the persistent problems that have 
plagued the center for many years. The most recent of 
these advisory committee reports on the state of educational 
statistics was explicit in their discussions of "persistent prob- 
lems" (Advisory Panel on Education Statistics, 1963): 

o long delays in the completion of many studies and 
reports; 

c difficulties in the reporting of basic information by the 
initiating sources; 

o less than adequate understanding of the needs of users 
and anticipation of new needs; 

o failure to utilize fully and effectively modern methods 
of collecting and analyzing data, including sampling; 
and, 

o lack of adequate planning, organizing, and scheduling 
of the work. 

It is somewhat sobering to contemplate the conclusion of 
the 1963 report of the Advisory Panel on Education Statis- 
tics, almost a quarter of a century ago, that "the indefinite 
continuance of these chronic problems is entirely unsatisfac- 
tory" (U.S. Department of Health, Education, and Welfare, 
1963:3). It remains so today, as concluded by the depart- 
ment's own Advisory Council on Education Statistics, which 
initiated the work of this panel because of its own concerns, 
as well as by many others who had made their views known 
to the council. Another important conclusion to be drawn 
from this history is that, whatever the reason, the advice 
and counsel that have been sought and obtained appears to 
have been ignored and forgotten. 

The most serious problems repeatedly cited to the panel 
concern th* quality and lack of timeliness of the data 
produced b> the center. These problems are widely known 
and have been openly expressed— in many of the more than 
50 papers prepared in connection with the center's ongoing 
redesign of its elementary and secondary data system and, as 
we noted, by the center itself as a major rationale for the 
development of its revised system of postsecondary data col- 
lection. 
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Quality 

The seriousness of this issue of quality is clearly illustrated 
by a quote from one of the redesign papers: "If the data 
continue to be as inaccurate in the future as they have in 
the past, all other issues are moot" (Drvid, 1985:143). The 
poor quality of the data is generally attributed to the fact 
that the data are collected, in large part, from administrative 
records maintained at the local level, which record "official" 
rather than "real" behavior; that the data are the product of 
diverse record-keeping systems that lack comparability in 
definitions and time periods; that the data provided to the 
center are at such gross levels of aggregation— such as for a 
state as a whole—as to seriously limit anyone's ability to 
check them for accuracy, consistency, and reasonableness; 
and that the data as published are at such summary levels of 
geography— such as a region-as to seriously limit their ana- 
lytical usefulness. 

Our review fonnd that, for the most part, the center 
lacks written standards to guide many, if not all, of its tech- 
nical activities, including those concerned with collecting 
data, monitoring contracts, and publishing reports. Many of 
the problems the center has encountered also lead us to con- 
clude that unwritten standards of performance either do not 
exist or are not known to the staff. Furthermore, the appli- 
cation of general statistical standards is highly variable 
throughout the organization and its products. In publica- 
tions, for example, technical detail is often lacking, error 
statements do not provide the required information, table 
columns or rows may not add to totals, table titles are not 
descriptive of the content, and headlines are not borne out 
by the data. In reviewing the background of the publica- 
tions, it is also clear that the comments of reviewers have 
not been taken into account. 

The effort to collect data on vocational education, as 
mandated by Congress in 1976, illustrates virtually every 
problem encountered in our review. The history of the 
Vocational Education Data System (VEDS), from its inception 
to its termination in 1983, is marked with failed attempts by 
a wide range of interested and involved parties, including the 
center, other parts of the department, states, and Congress, 
to establish, agree on, and accomplish realistic goals. 
Furthermore, the congressional mandate for VEDS failed to 
provide either staff or financial resources for the new effort, 
nor did the department shift resources to assist the center; 
consequently, the center had to consume large amounts of its 
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scarce resources on VEDS, to the detriment of other ongoing 
programs. 

Between fiscal 1978 and 1983, the center spent about $2.1 
million on start-up costs for VEDS and an average of 
$600,000 per year in operating costs. The costs to the 
states— and thus the eventual loss— was manyfold greater; the 
center estimated that state and local start-up costs were 
between $10 and $35 million and that operating costs were at 
$15 to $20 million per year. As noted, however, despite this 
considerable expense and effort, VEDS failed to meet its 
goals: it collected the full scope of planned data requirements 
only once in its five-year life. 

Furthermore, in late August 1983 the center notified the 
department that apparent problems had been identified in the 
data that had been collected and, in early 1984, the center 
confirmed its earlier statements and added that the VEDS 
data were unreliable and subject to serious misinterpretation. 
The center listed many reasons for the unreliability: lack of 
comparability among states, resulting in misleading national 
totals; year-to-year variability in the data, severely limiting 
the usefulness in trend analysis; within-state discrepancies, 
identified when data from VEDS were compared with data 
from independent sources; variability in collection methodol- 
ogy, reflecting poor management within states; a lack of uni- 
form reporting, resulting from difficulty in applying uniform 
definitions; duplicated counts—in a study of nine states in 
1981, duplicated enrollments ranged from 10 to 50 percent; 
changing universes, reflecting the changing composition of 
state plans; differential response rates, reflecting yearly vari- 
ations in the number of reporting units; missing data— a 
number of states were unable to provide the detail requested, 
particularly for financial data. 

In December 1983 the Office of Management and Budget 
disapproved the department's request for VEDS collection for 
1983-84. The center had begun the process of revising the 
program in August 1983; the sixth revision was submitted in 
September 1984; as of August 1986, collection of data on 
vocational education had not been resumed. Given this chain 
of events, one can easily agree with the thought expressed 
by a former administrator that "VEDS is a four-letter word." 

That the project failed is certainly unfortunate. Most 
damaging, though, was the high price exacted from the 
center— it bore the brunt from all sides, with substantial 
additional loss to its already fragile professional image and to 
its morale. 

VEDS, however, is not the only center program with 
problems. Response rates in the collection of the Common 
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Core of Data components from elementary and secondary 
schools have been far below any acceptable standard, yet 
little if any concern appears to have been expressed or any 
action taken until very recently. At one of its early 
meetings, the panel was astonished to be told that it was 
usual practice to have received less than half of the 
responses from the states by the cutoff data specified by the 
center, that six or more additional months might elapse 
before most of the delinquent states complied, and that some 
states just failed to comply. In the case of the most recent 
group of CCD surveys (those for the 1983-84 school year or 
later), not a single one of the six component surveys was 
completed by the due date. And two months later, the same 
situation existed—not a single survey had achieved a 100 
percent state return rate: three of the surveys were missing 
10 percent of the replies, one was missing 15 percent, and 
the remaining two were still awaiting replies from one-fourth 
of the states. And one year after the due date, three of the 
surveys were still incomplete. The panel was further aston- 
ished to learn that no procedures, such as organized follow- 
up, existed for dealing with noncompliance. Rather, the 
center sent a series of advance notices of a forthcoming 
inquiry to the states. Techniques for improving response 
rates, of course, have been standard procedures among data 
collection organizations for many years. 

In addition to lack of completeness, evaluation of results, 
investigations into possible sources of error, exposure to peer 
groups, record checks-standard operating procedures for 
statistical surveys--seem not to have been part and parcel of 
the center's operating philosophy, and their absence has con- 
tributed to the center's difficulties. The center's problems 
of data quality are further compounded by the lack of any 
specific individual or office with the assigned and explicit 
responsibility either for putting standards in place or seeing 
that they are carried out. 

The center's publications can also be faulted with regard 
to quality and accuracy. For example, a bulletin issued in 
April 1982 (NCES 82.3116) presented the results of a survey 
to determine the labor market experiences of students who 
completed vocational education programs in 1979. The meth- 
odological description is both so brief and incomplete as to 
be of little value to users. The role of the individual states 
in the collection of the data is confused, as is that of the 
education components within the states; response rates as 
given do not distinguish clearly between those for states and 
those for completers (a term that is never defined), and 
response rates for those states using sampling are not shown. 
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The weighted total of replies (which is erroneously labeled as 
respondents) appears in the body of the bulletin, but the 
number is omitted from the methodological discussion. That 
discussion does note the failure of three states containing 
half of one universe to participate in the survey, but this 
fact, or an appropriate caveat, are not noted in the body of 
the bulletin. A proportion noted in the text differs consid- 
erably from that which appears in the appropriate table. 
Finally, the data are presented to the unit digit (e.g., 
591,377), ascribing a wholly misleading and totally 
unwarranted degree of precision to the results. 

In similar fashion, data on enrollment in vocational educa- 
tion by provider and state appears in Table 3.2 of the 1983 
Condition of Education. In no fewer than nine states, the 
numbers of enrollees in vocational education in public secon- 
dary schools in 1979, as shown in this table, exceeded the 
total reported as enrolled in secondary schools, as shown in 
a table from another center publication, the 1979 Digest of 
Education Statistics. In yet another instance, review by an 
assistant secretary resulted in two pages of comments on a 
center report on nonresident alien enrollments in higher edu- 
cation, most of which addressed the issues of clarity, accura- 
cy, consistency, and ease of presentation; the report 
previously had been reviewed fully and approved at all levels 
within the center. 

The perception of the center is not helped when another 
part of the department openly criticizes a report issued by 
the center. In one case, the Office of Vocational and Adult 
Education (OVAE) challenged a report issued by the center in 
March 19S3 and based on expenditure data collected through 
VEDS. The NCES Bulletin, which was headlined "State and 
Local Expenditures for Vocational Education Continue Steep 
Decline" (National Center for Education Statistics, 1983b), 
does in fact represent an unsatisfactory piece of statistical 
presentation that passed through the center's review process 
without major change or criticism at any level. Both the 
basic work and the review were clearly inadequate. The 
headline refers to declines in expenditures; the data in the 
report refer to changes in the ratios of state and local 
expenditures to federal expenditures— a very different matter. 
The report also suffered from sloppy and inconsistent 
presentation, such as rounding errors and switching between 
current and constant dollars. The reaction of OVAE to the 
Bulletin, however, missed most of the inconsistencies and 
instead focused on the differences between early and revised 
reports from states. 
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The controversy did contribute to the center's initiation 
of an evaluation of the VEDS data quality (though no 
thorough process of checking back to the states was ever 
carried out), to the institution of formalized review processes 
within the department for center products and, ultimately, to 
the demise of VEDS. 

Of course, not all the center's work i:; bad, and the 
integrity of a statistical organization is seriously compromised 
when its advice, based on sound statistical procedures, is 
ignored. Such was the case when, in early 1984, at the 
urging of the American Association of University Professors 
(AAUP), the Secretary of Education directed the center, over 
the objections of both the center and the Assistant Secretary 
for Educational Research and Improvement, to provide AAUP 
with a tape that contained both incomplete and not fully 
edited data on salaries and tenure of staff in colleges and 
universities, collected as part of the Higher Education 
General Information Survey (HEGIS). Subsequently, an article 
in The Chronicle of Higher Education raised serious questions 
about the data and their release by the center. The final 
irony was the concern expressed at the top level when the 
center was subsequently accused of releasing the incomplete 
and incorrect files. 



One of the major concerns expressed by users of the center's 
data, and reflected in the decision of ACES to seek this 
review, has been the lack of timeliness in the data. This 
lack, which has resulted in inordinate delays in the publica- 
tion of important information, reflects a variety of causes. 
For example, poor planning or inadequate management of a 
survey process can result in an overly extended data collec- 
tion period, as in the case of CCD as noted earlier; clerical 
processing and data production requirements can be underes- 
timated; the complexity of activities can be ignored or not 
fully understood; time schedules can be allowed to slip; funds 
can be inadequate for the needs; and, finally, long delays can 
occur in the preparation of results for publication and in the 
process itself. The end result is that the data are less 
useful and users are frustrated. All of these factors have 
been present, either singly or together, in the center's 
efforts to produce its products. The result is that the most 
recent edition of the Digest of Education Statistics, published 
in December 1983, contains state-level data on student 
enrollment only through fall 1982, instructional staff data 
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only through fall 1 98 J , and finance data only through the 
1980-81 school year. As of August 1986, the latest published 
state-level data in each of these areas appears in the 1985 
edition of The Condition of Education and refers to 1983-84 
for school enrollments, 1981-82 for instructional staff, and 
1982-83 for school revenues and expenditures. The 1986 edi- 
tion of The Condition of Education^ originally scheduled for 
publication in June 1986, is now promised by the end of the 
year. 

The occasional bulletins issued by the center appear to 
have fared equally poorly. A Directory of Computer Tapes 
took six months to complete internal center review, following 
which an outside reviewer still found numerous problems in 
the publication. And. unfortunately, the time period between 
collection and publication remains long— one bulletin report- 
ing on data for 1982 was still in review in early 1985. 



Conceptual Obsolescence 

Another issue to be dealt with in the field of education sta- 
tistics is that of conceptual obsolescence. As pointed out by 
Bonnen (1977), two types of conceptual obsolescence occur in 
data: one, because of changes in the organization and 
framework of that which is being studied, in this case, the 
education system, and one, because of the changes in the 
demands on the system. Changing the design of data or its 
collection system-which the center is now attempting 
through its different redesign efforts—invariably involves a 
conflict between those who strongly favor the status quo in 
order to maintain comparability over time and those whose 
needs will best be met by the proposed changes. Resolution 
of conceptual issues is neither simple nor immediate. On the 
other hand, according to Bonnen (1977:399): "Failure to keep 
up with the changes in the policy agenda and in the reality 
of (education) leads to significant conceptual obsolescence, 
and the system begins to lose its capacity as an accurate 
guide for problem identification and solution or management." 
That this has been one of the concerns expressed about the 
center is borne out by the total of 59 articles, letters, and 
comments covering 785 printed pages and containing hundreds 
of recommendations from more than 50 experts, organizations, 
and agencies that the center received in response to a broad, 
public request to the education community to comment on the 
proposed elementary /secondary redesign program and to 
suggest ideas for improving the nation's store of data for 
education policy. 
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Funding and Staff Resources 



Finally, the center has long faced serious questions about its 
professionalism, staff competence, and management abilities. 
The management evaluation referred to earlier (U.S. Depart- 
ment of Health, Education, and Welfare, 1978), for example, 
listed ten problems, varying from a lack of response on the 
part of the center to policy needs, dissatisfaction with the 
timeliness of the products, general lack of responsiveness on 
the part of NCES to special requests or to users, to dis- 
agreement over the nature and extent of data analysis to be 
performed. 

In its response to the 1978 internal management review, 
the center noted two major problems that it believed wer? 
the cause of all other problems— funding and staff resources 
that were much lower than those of other statistical agen- 
cies. Whether the sole cause or not, what was true in 1978 
is equally so today: the center continues to be the smallest 
of the five primary federal statistical agencies in both dollars 
and staff. In fiscal 1985, its budget stood at $14.2 million 
with 134 authorized positions. In contrast, the National 
Center for Health Statistics, which, until the establishment of 
the Department of Education in 1980 was a sister statistical 
agency in the same department, had a budget of $42.8 million 
and 493 authorized positions. Both the center's budget 
(adjusted for inflation) and its staffing level have decreased 
more than 25 percent since fiscal 1978; other federal agencies 
primarily involved in statistical activities were cut by less 
than 1 0 percent (Congressional Research Service, 1 984). 
Whatever the reason for this treatment, it is quite clear that 
the center and its products have been seriously affected. 

The low level of funding for education statistics becomes 
even more startling when examined against the pattern of 
total expenditures for education. Comparing education and 
health, both highly decentralized program areas, we find that 
total government expenditures (including federal, state, and 
local) arc at similar levels, on the order of $150 billion per 
year for health against $180 billion for education (Bureau of 
Economic Affairs, 1984:Tables 3.15-3.17). But expenditures 
for federal statistical programs show enormous disparity: 
obligations for health statistics are estimated to be 3176.5 
million; for fiscal 1986, those for education statistics are 
estimated to be $18.9 million (U.S. Office of Management and 
Budget. 1985). In relative terms, then, the ratio of total 
government health expenditures to education expenditures is 
close to 1 to I, while the ratio of expenditures on health 
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statistics to those on education statistics is more than 9 
to 1. 

This extreme difference certainly raises some questions 
about the understanding of the need for and the commitment 
to high-quality education statistics, especially in light of the 
current public concern about educational quality. To cite 
another example, several years ago when the energy crisis 
was a major public issue, more than $100 million was pro- 
vided to the Energy Information Administration (EIA) to 
remedy the lack of energy statistics. Even in today's 
restrictive budget climate, the EIA budget for fiscal 1986 is 
approximately $60 million, or more than four times that for 
education statistics. It seems clear that, whatever the 
reason, the center has been seriously underfinanced. 

It would be quite simple if all the difficulties of the 
center could be blamed only on a lack of money. Unfortu- 
nately, as we have shown, the situation is otherwise.. The 
center continues to face three more basic problems: 

o it lacks a clearly defined mission and rale supported by 
agency leadership; 

o there is no agreement on an explicit framework as to 
what data should be collected and how they should be 
obtained or on the establishment of priorities within 
available choices to ensure that the center is success- 
ful in carrying out its mission; and 

o it lacks the technical capability and resources to do its 
job. 

The panel's reactions to the first two problems are addressed 
in Chapter 2; the iast concern is dealt with in Chapter 3. 
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Mission, Role, and Frame of Reference 



A continuing thread throughout the panels review has been 
the consistent lack of support for the center or, expressed 
another way, its seeming isolation from participation and 
involvement at the policy-making levels of the department. 
Although not a solution in itself, the center will benefit 
through having its role detailed more clearly in its mandated 
charter and, in turn, Congress and the Secretary of Educa- 
tion can and must contribute to the growth of the center as 
a respected, impartial voice on the condition of education in 
the United States. To this end, our initial recommendations 
in this chapter deal with the role, mission, and responsibili- 
ties of the center; the second set of recommendations con- 
cern data collection strategy and, in a general way, data 
substance; our final recommendations concern the center's 
frame of reference. 

The panel's recommendations establish a role for the 
center and recognize its importance in reaching out to policy 
makers and to its user community and involving them in dia- 
logue on data needs. The center's recent efforts in this 
regard in connection with the redesign of the elementary/ 
secondary data collection system are to be applauded, but 
this process must become an inherent part of ihe censer's 
activities and not an occasional activity resulting from the 
concerns and thus the pressures of concerned data users. In 
this connection, we provide some general guidance in terms 
of needed program content and program direction that 
departs from the state-oriented collection of administrative 
data and points to the need for sample-based national data 
that are essential for understanding the functioning of the 
nation's educational system. 
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MISSION, ROLE, AND RESPONSIBILITIES 

It is essential that support for the center and its indepen- 
dence be clearly articulated by both Congress and the rele- 
vant departmental officials and that the appropriate actions 
be taken to demonstrate the full extent of the support. His- 
tory shows that the center has seldom, if ever, shared its 
knowledge and expertise, even in the extermination of its 
resource needs, either with the Office of Management and 
Budget or with the appropriations subcommittees of Congress; 
the center invariably has been represented by other entities. 
Thus, its needs and their justification have been presented by 
others, not the optimum way for the center to gain respect 
and credibility. In the panel's view, the department must 
provide the center with the opportunities to be seen and to 
be heard, to have the opportunity to respond directly and 
openly to challenges and to criticism, to be held responsible 
for what it says and what it does, to understand through 
direct colloquy what is expected of it, and to understand why 
and how its requests have been disposed of. It is also 
essential that the center be viewed as objective, unbiased, 
and nonpartisan in carrying out its mission. 

Even beyond support, the center requires recognition of 
its role in participating in the determination of data needs 
and of its capabilities in determining if and how the needs 
can best be met. Furthermore, the center must be recog- 
nized as the key source for compiling and issuing data, 
including the release of key indicators and especially for 
assessing progress i.n education among the states. Given the 
history of the center, the panel believes that its position 
would be strengthened by broadening and detailing its man* 
date and by positive actions on the part of Congress. We 
propose several first essential steps in the process of refo- 
cusing the center. 

o The panel recommends that the mission of ;he Center 
for Statistics, as stated in Section 406(b) of the 
General Education Provisions Act, as amended (20 
U.S.C. 1221e-l), be strengthened to clearly establish 
and define the role of the center in assisting the Sec- 
retary of Education ; .n determining the data needs for 
assessing the condition of education in the United 
States, and for accepting responsibility and account- 
bility for ensuring the availability of the necessary 
data. 
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o The panel recommends that Congress demonstrate its 
support for the center and its mission through its bud- 
get actions, through its requests for informed statis- 
tical advice, through requests for direct testimony from 
center officials, and through its calls for education 
data. 

o The panel recommends that the Secretary of Education 
demonstrate strong and continued support for the cen- 
ter and its mission through department budget requests, 
public statements, and in appearances before Congress. 

o The panel recommends that Congress and the secretary 
recognize and state their support both for the nonpar- 
tisan nature of the center and for its statistical inde- 
pendence. 

o The panel recommends that the secretary designate the 
center as the functional agency responsible for coor- 
dination and technical review of all data collection 
within the department. 

o The panel recommends that the secretary designate the 
center as the focal point of releasing statistical infor- 
mation on education. 

o The panel recommends that the center assume account- 
ability for the process that leads to the determination 
of the content of the data to be collected. 



Given the ongoing debate on education, it is quite clear that 
the center needs to make substantive changes in programs 
and in overall strategy. We believe the following recommen- 
dations are both basic and essential to providing the under- 
pinning for the more difficult steps to follow. 

o The panel recommends that the leadership of the cen- 
ter develop mechanisms to assist policy makers in 
determining their data needs and that the center con- 
tinue to involve its users in dialogue to determine the 
most relevant and appropriate content of its Jata col- 
lection efforts. 



CONTENT AND DIRECTION 
OF DATA COLLECTION PROGRAMS 
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o The panel recommends that the center continue the 
compilation of program, staff, and financial data from 
the states, but that it undertake an analysis, jointly 
with representatives from the states, to ensure that 
the present program meets both center and state 
requirements for usefulness, relevance, quality, and 
reliability. In addition, the content of the data on the 
condition of education, originating with the states, 
should be monitored periodically with t^e intent of 
improving the content and thus its usefulness to the 
center and the states. 

o The panel recommends that the center initiate a 
sample-based program of data collection focused on 
individual classrooms and students, designed to facil- 
itate better understanding of the relationships between 
educational inputs, processes, and outcomes. The sam- 
ple would not necessarily be able to provide adequate 
estimates for all individual states, although i: should 
be organized so that states could choose to augment 
the sample to provide state data. 

o The panel recommends that sample-based data on edu- 
cational inputs, processes, and outcomes be made 
readily available, on a timely basis, to all interested 
users in the form of public-use tapes, with appropriate 
masking of characteristics to preclude the identification 
of individual schools, teachers, or students. The panel 
believes that the usefulness of center data would be 
greatly enhanced if its data become widely available as 
a resource for both policy and research use. 

o The panel recommends that the center continue to 
explore the inclusion of longitudinal features in its 
sample-based survey data and that, as a m/Jiimum, 
small-scale longitudinal studies be mounted regularly as 
part of the center's data collection efforts. 



ESTABLISHING A FRAME OF REFERENCE 

Every organization in carrying out its mission and role oper- 
ates within a set of guidelines, a frame of reference against 
which it measures and weighs what it should be doing, what 
it is doing, and, finally, how well its completed tasks meet 
its criteria. In many cases, the framework is fully articu- 
lated and spelled out in directives or other pronouncements 
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of the organization; in other instances, the history of perfor- 
mance stands as the object illustration, and little if anything 
is found in written, explicit form. The center appears to fall 
very loosely into the latter group-the panei was unable to 
find a clearly defined conceptual framework that has served 
as the guide for the center. In fact, we would argue that 
the history of the center suggests that it has operated pri- 
marily in a reactive mode, facing each problem or challenge 
somewhat independent of the one before, and that its frame- 
work or operating model has neither been documented nor is 
readily discernible through observation or from conversation 
with staff. 

o The panel recommends that the center develop a con- 
ceptual framework for organizing its program and for 
setting priorities in light of available resources. 

Such a step will provide a basis for planning and a basis for 
review and evaluation of problems. This section presents the 
panePs views on some aspects of a framework and some 
desirable characteristics of a data col? ration system. 



A model for the center's data collection activities should be 
characterized by its ability to meet two major objectives that 
the panel s~es as essential to a system of educational statis- 
tics. First, almost all of the data collected by an educa- 
tional statistics agency snould reflect the interplay of educa- 
tional input;, educational resources, and educational out- 
comes. Longitudinal data are required to trace the influence 
of resources applied at one point in time to outcomes 
observed at a subsequent point in time. Second, a data col- 
lection system must be guided by the needs of users and 
reflect the joint concerns of both the research community 
and the policy community. 

The role of the research community is to ensure the 
collection and reporting of data most relevant to analysis of 
educational processes and outcomes. The involvement of the 
research community also contributes to the continuing evalu- 
ation of the adequacy of the data, both in coverage and in 
statistical quality. The policy community sets the agenda for 
data collection and thus establishes the needs for data rele- 
vant to policy decisions (even if, at that time, the data have 
no well-defined role in an existing model). 



A Model for Data Gathering 
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The center's data collection efforts should comprise ele- 
ments that can be classified along several dimensions, such as 
serving primarily analytic uses, in which they form an essen- 
tial element in an analytic model of the educational process, 
or as serving primarily descriptive uses, in which they inform 
users about the size or distribution of a particular character- 
istic of the educational system. Data elements can describe 
micro-environments, such as activities in a particular school 
or classroom, or they can describe macro-environments, such 
as all schools or classrooms along a particular dimension. 
Data can be cross-sectional, such as describing the state of a 
particular education variable at a specific time, or longitudin- 
al, describing a process of change that takes place over a 
given time period. Finally, data can relate to any of five 
levels of the educational system-preschool, primary or ele- 
mentary school, secondary school, postsecondary school, and 
education at work. 

As we have noted earlier, a data system must be 
perceived as an evolving collection of measures in which 
design of the system not only reflects existing or past inter- 
ests and needs of the education community, but also attempts 
to anticipate future interests. As understanding of relation- 
ships among inputs, processes, and outcomes changes with 
new research knowledge, and as policy needs change, mea- 
sures of central importance at one stage of development of a 
system are likely to be replaced by other measures. 

This process of change will be enhanced if opportunities 
for analyzing the data are regularly made available to outside 
users as well as to center staff. The quality and utility of 
data are tested through the close scrutiny of data analysts 
and researchers trying to understand the effectiveness of the 
educational system or the relationships between inputs and 
outputs or the mechanism by which various system inputs are 
generated. In similar fashion, use of data by policy makers 
and the public serves as a test of both quality and utility. 



Implementing a Framework 

Fulfilling the center's mission-to collect and disseminate 
statistics and other information related to education in the 
United States and other nations and to design a system of 
statistics that not only satisfies current information needs 
but also anticipates the f uture information needs— requires 
that the center improve its approach toward setting priorities 
and allocating resources. In so doing, it must make clear 
how what it collects augments information available on edu- 
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cation in the United States from other sources and how, col- 
lectively, the available information fits within a structure 
that also makes clear what information is not being collected. 

One might begin this process by exploring the issue of 
the outcomes of education. For example, answers might be 
sought to such questions as the extent to which students 
learned what they were taught; the extent to which students 
were prepared for the next steps— further education or work; 
and the extent to which students have accumulated skills 
that permit them to function as adults. Currently, criticism 
is heaped on the meager data currently available on student 
outcomes, whether they be achievement rates, reading accom- 
plishment, dropout measures, or conclusions on labor-market 
transition success-all are considered inadequate. Taking the 
lead in identifying and seeking agreement on appropriate out- 
come measures and devoting imaginative and innovative 
thinking to possible data collection programs would certainly 
be a forward-looking and much-needed step for the center. 

Ideally, the center should provide information that allows 
examination of the differences among students and school 
systems and other potential influences on outcomes. These 
influences include the available resources-funding, qualifica- 
tions of teachers and administrators, curriculum quality, 
characteristics of the setting in which education takes place 
(e.g., policies, organizational structure, physical facilities), 
and context-characteristics of the students and their fami- 
lies, (e.g., cognitive and social skills, and public attitudes.) 
For each potential influence, the statistics system must try 
to describe the parameters of the potential influences (e.g., 
the knowledge, skills, and dispositions of teachers) as well as 
to model the origins of those characteristics (e.g., studies of 
teacher education, recruitment, and selection). Because each 
of these influences is complex and represents a cost, the 
center must be responsible for establishing the priorities to 
determine the area to be explored and developed, for deter- 
mining the information to be collected, and for providing the 
logic behind the choices. Although this brief discussion of 
educational outcomes and of potential influences is written 
largely from the perspective of describing the education that 
is provided in schools and colleges, the general framework 
also applies to preschool education, both formal and informal, 
and to education provided in the work environment. Ideally, 
s system of educational statistics would also be capable of 
describing transitions from education at one level to educa- 
tion in another. Key points of transition are from preschool 
to elementary school, from elementary school to middle 
school, from middle school to high school, from high school 
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to postsecondary education or to work, and from postsecon- 
dary education to work. These points of transition raise 
questions concerning how well the educational outcomes from 
one level serve as appropriate inputs for productive educa- 
tional experiences in another level. They also raise questions 
about the continuity of learning experiences and the effects 
of continuity or discontinuity on educational outcomes. 

Because the mandate for education in the United States is 
broad, because the U.S. educational system is complex, and 
because the influences on educational outcomes are so little 
understood, an ideal system of education statistics goes well 
beyond the resources now available to the center. It is 
especially important, therefore, that the center design and 
maintain its system of education statistics in a public and 
open forum, giving full access to users, policy makers, and 
researchers. The center is to be commended for having 
recently moved in this direction by involving a wide variety 
of constituencies in assessing the nature of its current work 
and in soliciting suggestions for future work. At the same 
time, it must be recognized that a system of education sta- 
tistics that supports descriptions of trends over time is 
absolutely essential. Thus, responsiveness to current need* 
must be tempered with the requirement that a core set of 
data be maintained over time. 



40 



3 



Designing a Program 



Given our view that the Center for Statistics requires a fun- 
damental change in its methods of operation, we stress once 
again the need to view our recommendations as a unified 
program. It is true, particularly in the areas covered in this 
chapter, that individual recommendations address what appear 
to be unique problems, but we believe the necessary and 
desired transformation of the center will occur only if all of 
the panel's recommendations are considered as a whole. The 
appropriate analogy is that of a complex machine that has 
many interdependent parts: fixing what appears to be one 
deficient part does not result in a workable machine; further- 
more, fixing or replacing parts one at a time is both ineffi- 
cient and counterproductive, since action (or its lack) in one 
area directly affects the progress in others. 

Although we recognize that some additional funding will 
be required to fully implement our recommendations, we have 
not proposed a solution that rests solely on immediate, large 
infusions into the center. In the short run, the center will 
require modest additional support for the staff needed just to 
carry out its present program; however, we believe it unrea- 
sonable (as well as unwise) for the center either to request 
or to expect further support in order to accomplish the pan- 
el's recommendations without first having demonstrated a 
commitment and a strong beginning towards improvement. 
Although difficult choices may be required of the center, it 
is our view that much can be done within its existing budget 
and staffing level. 

To this end, as needed, the panel strongly urges that the 
center plan and be prepared to divert resources or curtail 
existing programs or their scope in order to implement and 
accomplish the panel's recommendations. Given the wide 
range of possible actions and choices before the center, the 
panel does not believe it is the appropriate body to suggest 
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cither the specific programs to be curtailed or which 
resources might be diverted; rather the panel emphasizes the 
need for the center to develop a mechanism to ensure that 
priorities and choices are openly discussed and chosen and 
made known to users and other interested parties. 

This chapter is concerned with the realities of carrying 
out the program of the Center for Statistics. We begin by 
reviewing the role of the Advisory Council on Education Sta- 
tistics, and then provide guidelines and recommendations for 
improving each of the areas involved in the production of 
data, including staffing and budget, quality control, design, 
collection, processing, and publication. 



THE ADVISORY COUNCIL 

The Advisory Council on Education Statistics was established 
by statute in 1974, and the initial members were appointed in 
mid-1975. It meets a minimum of four times a year, and its 
members serve three-year terms. The Assistant Secretary for 
Educational Research and Improvement serves as the presid- 
ing officer. In that the ACES charter directs it to "review 
general policies for the operation of the Center," it follows 
the practice of virtually all advisory groups—namely, to 
advise. What distinguishes ACES and sets it somewhat apart 
from other groups is the part of its charter that states that 
the Council ". . . shall be responsible for establishing stan- 
dards to insure that statistics and analyses disseminated by 
the center are of high quality and are not subject to politi- 
cal influence." 

The usefulness of any advisory group, of course, is 
dependent on a number of different factors, including the 
stature of its members and the extent of their competence in 
the topics under review; the types of information provided 
the group, the problems brought to it, and the cooperation it 
receives; the interest, concern, and responsiveness of those 
to whom the group provides its advice and counsel; and the 
degree of understanding and reaction on the part of the top 
management in the activities and actions of the group. 
Overriding these factors, of course, is the fact that any such 
group has neither operating nor policy authority. 

The panel's review reveals that ACES has had a checkered 
past. In its early incarnation, its members indeed were tech- 
nically oriented and extremely competent to review operating 
policies and comment on how the center carried out its mis- 
sion. Later, the membership changed to reflect a broad 
variety of experience, but not in the collection of education 
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data or in topics necessarily related to the concerns of the 
center. As such, ACES appears to have become considerably 
less relevant and useful to the center over time. Moreover, 
ACES does not appear to have received any reaction to the 
concerns that it has raised in its reports to the secretary or 
to the Congress. Furthermore, for a variety of reasons, 
including the budget problems already noted, there appears to 
have been little or no reaction on the part of the center 
itself to the concerns raised by ACES. This lack of interest 
also appears to be fostered by having an assistant secretary, 
rather than the administrator of the center, sit as the 
presiding officer. 

With regard to the mandate to establish standards, ACES, 
in its first annual report prepared in March 1976, expressed 
its support for the addition of a senior mathematical statisti- 
cian to the staff and indicated it would return to discussion 
of the issue at future meetings. Successive reports dealt 
with one or another aspect of this issue and, as noted ear- 
lier, it was ACES that initiated the present study because of 
many concerns, including its perception of the continuing 
lack of standards within the center. As the panel was com- 
pleting this report in mid-1 986, it is informative to note that 
for the previous year or so the center had been seeking (but 
had not yet found) a principal statistician to assume respon- 
sibility, among other areas, for "the setting of standards and 
developing ways to ensure they are known and observed sys- 
tematically in the Center" (letter from the dh ctor of the 
center). 

With a goal of once again ensuring the presence of a 
range of technical competence, the present leadership of 
the department replaced the membership of ACES. It is 
obvious that if ACES is to be of any value to the center, it 
must consistently represent technical competence, both opera- 
tional and substantive, and the panel urges that this be the 
key criterion for member selection. Nevertheless, it is the 
view of the panel that ACES alone cannot provide the wide 
range of technical advice that the center seeks and requires. 
Furthermore, we strongly believe that it is wholly inappro- 
priate for ACES to be held responsible for establishing stan- 
dards, a mandate that it never was able to implement. As 
we noted earlier, an advisory group can advise and provide 
an oversight function, but given ACES's lack of authority and 
intermittent operation with no staff, it should not be 
expected to establish standards. Responsibility for establish- 
ing and implementing standards-and ensuring that its prod- 
ucts are of high quality and not subject to political influ- 
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ence-must and can only be the inherent responsibility of an 
operating entity, in this case, the center. 

o The panel recommends that Congress n.udify the 
mandate of the Advisory Council on Education Statis- 
tics to a rele of technical review and oversight, rather 
than responsibility for the establishing of statistical 
standards. 

o The panel recommends that the Secretary of Education, 
in support of the center, provide timely written 
response to the concerns raised by ACES in its annual 
reports to the secretary, and that Congress periodically 
hold hearings at which both the secretary and repre- 
sentatives of ACES provide testimony on the status, 
problems, and progress of the center and the concerns 
and advice of ACES. 

o The panel recommends that members of ACFS be 
selected for their expertise and competence in areas 
directly related to the center's program. 

o The panel recommends that the center establish one or 
more technical advisory groups, in addition to ACES, 
to meet at least twice per year to review, comment, 
and advise on the methodology used in the conduct of 
its work, and to provide the director with objective, 
outside, independent review of the technical capability 
of the center's staff or of its products. As required 
by its changing program needs, the center also should 
establish ad hoc advisory groups representing unique 
technical and subject area skills. 



IMPROVING DATA QUALITY 

Statistical Standards 

The essential underpinning of data quality is the establish- 
ment and implementation of standards. Standards are the 
norms of the research community as to what constitutes ade- 
quate practice, and they provide mechanisms for the public 
exposure and censure of deficient practice. As noted earlier, 
ACES, by law, has the responsibility for establishing the 
standards which both guide the center's work and ensure that 
its products are free from political influence. In fact, this 
responsibility can only be carried out by the center. 
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Our earlier exposition of the many different p 
faced by the center also serves to detail what the 
must do to improve the quality of its data and its p 
With regard to standards, we note that they have rc 
and meaning only when they are required and expc 
staff as a normal part of any task. Staff must unc 
the need for standards and must be technically comp< 
implementing them. Data quality is the product of < 
ment, competence, and pride. The center should n 
not-accept less from its staff. It should also intera 
those who provide its data to ensure that they, i 
understand what is required of them and why, and it 
assist 'he states and local school districts in carry 
their functions in a satisfactory and timely manner. 

The panel notes that certain of the center's pre 
for example, the longitudinal studies-are currently 
nized by users as providing high-quality, valuable, anc 
needed data. In some cases, the products have beer p 
by staff; in other cases, the center has been very su< 
in and is benefiting from the selection of contractors 
reputation, operating philosophy, and accomplishments 
demonstrate high standards. Simply put, the center i 
requires more of each. 

o The panel recommends that the center develop, \ 
disseminate, and implement standards to guide t 
duct of all phases of its work, from developn 
objectives through collection, follow-up, and pre 
and including the preparation, review, analys 
publication of results. 

o The panel recommends that the center establ 
Office of Statistical Standards and Methods an< 
expeditiously to recruit and appoint a chief stat 
at the level of assistant director, with respon 
for the establishment and maintenance of sta 
standards throughout the center. 

o The panel recommends that the center, in conce 
appropriate state and local education agencies 
and representatives of institutions of higher edu 
institute the development and publication of u 
definitions and their continuing review, to ensi 
collection of consistent data, such as between di: 
LEAs and states. Such information should be 
available regularly in reference handbooks. 
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o The panel recommends that the center undertake a 
continuing program of evaluation of its methodology 
and its programs. 

o The panel recommends that the center disseminate 
widely the results of its methodological research. Pub- 
lications in professional journals and presentations at 
conferences should be strongly encouraged. 

o The panel recommends that the center require that its 
reports contain information on definitions used, in stud- 
ies and a description of the data collection methodol- 
ogy, provide a measure of reliability, if applicable, and 
discuss possible sources of error. 



Redesign of the Elementary/Secondary Data Program: 
An Example 

Quality is a goal of all statistical organizations; recognizing 
and measuring it are the tools by which the goal is accom- 
plished. The approach is called quite simply "quality con- 
trol," and error is the villain. The many possible sources of 
error in statistical data and what can be done to expose, 
measure, and control them are well known and expounded on 
in textbooks on quality assurance. It is important to recog- 
nize that errors can arise from many sources at every stage 
in the collection, processing, and analysis of data and, fur- 
thermore, that some error is unavoidable (such as sampling 
error when conducting a sample survey, or recording error 
when using administrative records as the source of data). 
When designing a system to provide statistical date, all the 
potential errors need to be considered as well as the costs 
involved in reducing them. 

The current effort to redesign the. elementary/secondary 
data system provides a clear example of how these and many 
other issues surface and must be dealt with when fundamen- 
tal changes ?.re proposed. For example, should the common 
core of data be collected as at present-that is, through a 
complete enumeration of all school districts using, for the 
most part, administrative records-or should the data or some 
variant be obtained through a sample survey approach? On 
one hand, a sample survey approach-in which a sample of 
LEAs would provide a sample of schools and, in turn, a sam- 
ple of pupils, teachers, and officials in these same schools— 
as proposed to the center by a panel consultants (Hall et 
al., 1985)— is seen as providing more timely and more accu- 
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rate data. Proponents argue that it would permit greater 
control of the entire data collection process, provide for 
analyses of relationships within and across major areas of the 
education system, and result in reduced burden on respon- 
dents and that such a system also would provide longitudinal 
capabilities to relate outcomes to earlier experiences. Even 
this minimal list of potential benefits seems to lead to an 
obvious conclusion. On the other hand, the costs of imple- 
menting such a system would be great. Critics note that the 
basis for assuming the data would be either more timely or 
accurate is not firmly established and that the problems of 
integrating or linking the diverse elements would be many 
and difficult. 

Although the current system raises serious concerns about 
the presence of large response and other types of nonsam- 
pling errors, the Common Core of Data program is institu- 
tionalized and has been in operation for a number of years. 
It has been adapted by the individual states to provide for 
many of their continuing needs for data, as well as being 
financially supported virtually in full by the states. In the 
words of the National Education Association (1985:738): 

The Core represents the most basic data series within 
the NCES. It enables assessments of what was, what 
is, and what will be in a statistical sense. Annual 
updates to Core surveys provide basic statistical infor- 
mation on public schools, their pupils, personnel, and 
finances . . . the Common Core may represent the most 
heavily used series of public school statistics. The 
Core is the cornerstone of educational information in 
the United States. No other public or private institu- 
tion collects and maintains public education data to the 
extent that NCES does via the Core. 

The choice of design also is often complicated by the 
multipurpose nature of statistical data--thfi preferred design 
for one purpose may not suit the needs of another. Thus, 
were all other factors equal, collecting common core data on 
a sample basis with controlled data collection almost certain- 
ly would be the preferred method for national and regional 
estimates; however, this method may not adequately serve the 
need for local area estimates. And if local school officials 
are unwilling to support the desired approach, that fact may 
outweigh all other factors. 

On balance, however, ths panel favors and supports a 
move to collect elementary and secondary data through a 
survey-based approach as a long-range strategy. Our under- 
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standing of the present character of the center's relationship 
with its data suppliers and the center's limited experience in 
carrying out such a survey lead us to conclude that abrupt 
and wholesale replacement of the existing system would entail 
an unwise and unnecessary risk. The center must not make 
the mistake of moving precipitously without undertaking the 
necessary testing and development, without first demonstra- 
ting to the states and local areas, as well as to itself, how it 
will deal with the issues that have been raised, what the 
costs of the program will be, and how and why a survey 
approach will produce consistent, accurate, and reliable data. 
The center also has a responsibility to the states to ensure 
that there is agreement on what their respective responsibil- 
ities will be under the new program and how the unique 
needs of different states will be met and supported. Thus, 
the panel supports serious study, development, and testing of 
an integrated system based on sample surveys, but urges cau- 
tion, judgment, open discussion, and continuous review with 
all parties prior to the adoption of a new system. Above all, 
it urges that there be adequate testing to confirm the feasi- 
bility, probable cost, and the center's ability to accomplish 
the objectives of the program. In fact, the center may wish 
to overlap some survey-based components with the present 
CCD program as a learning experience, to demonstrate its 
feasibility, and to provide comparative data. 

o The pane! recommends that the center continue to 
explore most carefully the appropriate balance between 
the continuing use of state and local administrative 
record sources and the use of sample surveys. 
Although supportive of the use of sample surveys, the 
panel does not believe that the center has fully 
resolved the issues involved in shifting the elementary/ 
secondary data collection system from one based on 
administrative records to one based solely on an inte- 
grated sampling survey approach. The panel endorses 
the efforts of the center in seeking new approaches to 
accomplishing its mission and suggests that small-scale 
testing within one or a few states be undertaken to 
establish procedures, define problems and develop solu- 
tions, refine the approach and, overail, assess the 
feasibility and cost of the survey proposal. 
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Detecting and Reducing Error 



Another major reason for needing information on the magni- 
tude of errors from different sources is in order to be able 
to provide data users with an assessment of the quality of 
the statistics produced. Sophisticated users will demand 
information on quality, but unsophisticated users may not. 
However, even without specific demands, the center has the 
responsibility not only to produce education statistics but 
also to inform its data users of the limitations of those sta- 
tistics. 

o The panel recommends that the center routinely make 
its users aware of the quality of the data and the 
limitations inherent in their use. 

The nature of error, its many sources, and its measure- 
ment need to be understood as a proper basis for decision 
making with regard to the best way to collect, process, and 
publish data. It is only realistic to accept the inevitability 
of some error; at the same time, much of the error in a data 
collection process can be reduced by an effective and willing 
statistical agency. 

The fundamental requirements, of course, are a general 
inquisitiveness on the part of the staff toward the entire 
process and toward the numbers collected, a distrust of the 
information conveyed by a sequence of digits, and an ever- 
present realization of the ease with which error can be 
introduced in data. Together, these factors combine to 
create an atmosphere in which efforts to minimize error 
become basic to the planning and execution of the data col- 
lection process and, later, in which the data as collected are 
examined for error and appropriate actions taken. 

There are variety of opportunities to exercise an error 
detection and correction process. The use of appropriate 
control procedures provides the ability to monitor the opera- 
tion on a timely br.sis: sample checks made throughout the 
data collection process ensure that the operations are being 
conducted as specified; internal consistency checks are done 
when the data are obtained; for data that are collected peri- 
odically from the same units (e.g., schools or universities), 
present and past responses are compared to identify anoma- 
lies. Similarly, independent checks against record sources 
serve as a valuable quality control measure. Reviews by sub- 
ject-matter specialists point out analyses that compare statis- 
tics that are not directly comparable, noting the need for 
additional variables in regression models. 
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All of these quality control measures, if they are to be 
successful, require that they be fully documented-both the 
procedures and the results must be available on a timely 
basis during the process. In addition, historical records must 
be maintained and readily available to guide in the planning 
for new projects with similar characteristics, to assess the 
effectiveness and efficiency of the quality control mechanism 
over time, as a teaching device, to correct faulty memories 
and, above all, to be exposed to critics and supporters alike, 
for comment and suggestion. 

Particular care is needed in checking final estimates prior 
to publication. Errors made in the analysis and in the prep- 
aration of estimates for publication are potentially the most 
damaging. Results can be compared with those produced 
from other sources to see whether they are reasonably con- 
sistent; if not, an explanation of the discrepancy can be 
sought. Checks on data that are part of a time series can 
be made by examining the reasonableness of the changes that 
have occurred since the previous data collection. Errors also 
are less likely to be introduced into a publication if those 
responsible for its preparation are knowledgeable about its 
subject matter and thus able to assess the reasonableness of 
the data as they complete the analyses. At the most basic 
level, users expect that published data have been checked to 
ensure that detail adds to totals; that figures found in the 
text are those in the tables; that table titles reflect table 
content; and that highlights reflect the data. The panel has 
reviewed a number of examples of center publications that 
fail to meet these criteria and finds such performance by a 
statistical "agency to be unacceptable. We believe that the 
Center for Statistics has a responsibility to ensuic :!iat its 
reports are a model of statistical integrity—complete, correct, 
and consistent. 

Our recommendations deal in part with this lack of sen- 
sitivity to the ^biquitousness of error and the responsibility 
of a statistical agency and its staff to track down, diagnose, 
and correct as much of this error as possible before dissemi- 
nation. 

The usefulness of the publications produced by a statis- 
tical agency also is affected by the success of the staff in 
diagnosing and identifying the sources of error, as well as in 
measuring the amount of error that cannot be corrected. 
Interaction with data users helps focus error detection 
efforts on the areas that will benefit users the most. The 
openness and frequency of interaction of a statistical agency 
with its data providers helps to promote the establishment of 
standardized definitions and data collection procedures among 
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the providers. Thus, the relationships of the center with its 
communities of data providers and data users become an 
important and essential element in the achievement of qual- 
ity. 

o The panel recommends that the center foster a climate 
of openness in dealing with all elements of its consti- 
tuency, including other parts of the department, other 
government agencies, Congress, providers and users of 
its data, and its professional peer groups. 

o The panel recommends that the center initiate a 
comprehensive program to assess and improve, where 
necessary, the quality, consistency, and reliability of 
data obtained from state and local agencies, from in- 
stitutions of higher education, and from other sources. 
Meetings, discussions, visits, and technical assistance, 
including the training of local and state officials on 
what is required of them and their systems, must com- 
prise a major component of the program. 

o The panel recommends that the center establish stan- 
dards for review of data releases and institute pro- 
cedures to effect them. 



TIMELINESS 

It is essential that any system of collecting education data 
recognize, reflect, and react to the issue of timeliness. 
Timeliness has two dimensions: collecting timely data and 
providing results in a timely fashion. Analysis becomes vir- 
tually useless if long-term gaps are allowed between collec- 
tion points, particularly for topics that may show large 
directional shifts in relatively short time frames. Even more 
important, failur? to anticipate or recognize a shift in 
emphasis or need inevitably results in a lack of timely and 
relevant information when the policy agenda takes one of its 
many turns. 

An example of such a lack is the case of teaching and 
teachers. With the publication of A Nation at Risk (National 
Commission on Excellence in Education, 1983), these topics 
emerged as fundamental issues of concern, and the need for 
data was sudden and immediate. Unfortunately, information 
on the number of teachers and other professional staff— 
which we would think would be an essential element of any 
continuing data system-was last collected at the elementary 
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ard secondary level in 1979-80. Data on minority teache 
even more archaic, having last been collected in 1968. 
the postsecondary level, the most recent data are for 1 
hardly informative or useful for the concerns of the 
1980s. Turnover rates among teachers provide yet anc 
example, having last been collected in 1969. 

Although hindsight is certainly easier than foresight, t 
examples clearly illustrate several key Tacts. First, had 
center's planning and decision process been guided b> 
organizing framework that required a balanced appn 
across and description of the phases of education-in] 
processes, and outcomes— at least minimum information 
teachers and teaching would have been anticipated and a 
able as a regular component of the center's data collec 
program. Second, an appropriate system of education st; 
tics must strive to anticipate the needs, both of researc 
and the policy community. Finally, it is important that 
periodicity of data being collected be reviewed regularl 
ensure that trends are being monitored on an appropi 
time continuum. 

Of the need to provide data in a more timely fash 
there can be no argument. Too often, the center has fc 
itself embarrassed by an excessive and unacceptable time 
between collection and release of data. Perhaps there is 
better example than the 1985 Condition of Educe 
(National Center for Education Statistics, 1986), which st 
its objective as providing "accurate and timely statistics" 
"the center's major response to its continuing mandate 
to report full and complete statistics." But the vast ma jo 
of the data presented in this report are for 1983 or ear 
and in many cases the 1983 data are marked as "prelimin 
or "projected." As to the references to "accurate" and * 
and complete," one needs only to note again the example 
Chapter 1 or to refer to the Synthesis of Invited Papers 
the Elementary/Secondary Education Data Redesign Pro 
(Silverman and Taeuber, 1985c). Some of the concerns \ 
highlighted in an article by Jonathan Friendly in the 1 
York Times of March 11, 1986: "Last summer, for exair 
researchers asking the center for state-by-state totals 
school enrollment had to make do with 1983 figures, 
they were luckier than the people who needed statistics 
teacher training and salaries: the center's most recent n 
bers for that dated to 1980 and 1981, before most of 
state measures to improve education were enacted." 
timeliness of the center's performance has been abysma 
must move aggressively and rapidly to overcome both 
problems and its poor reputation. That the center appear 
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be facing up to its critics and their criticisms is to its 
credit, but the proof of its intentions will lie in the extent 
and speed of change and the degree of ccirimitment to 
improvement, rather than in repeating the words in its statu- 
tory mandate. 

o The panel recommends that the center explore and 
develop approaches for expediting response from par- 
ticipants in its studies and techniques for producing 
usable advance results; use management information 
systems for monitoring progress through all phases of 
data collection and reporting; and establish procedures 
for taking prompt and appropriate actions when 
required. Similar techniques and monitoring controls 
should be required as integral components of work 
done by contractors to the center and should be fol- 
lowed closely to ensure prompt adherence to schedule, 
with penalties for failure to accomplish work on time. 



The presence of competent, experienced, and dedicated staff, 
with the required mix of disciplines and expertise, is basic to 
the success of the center. Given its mission and responsibil- 
ities, its needs range across a broad spectrum of talents 
required in collecting and producing education statistics, 
ranging from organizational leadership and management skills 
to clerical and semiprofessional support, and at a minimum, 
the expertise necessary to collect, process, analyze, and 
present statistics, whether collected from existing records or 
obtained directly through surveys or censuses. Furthermore, 
the staff must have the capacity, capability, and interest to 
prepare and publish regular analytic reports based on the 
data. Such efforts develop both a substantive and technical 
competence and insight on the part of the staff, which 
result, first, in better data, and, second, in better under- 
standing of the meaning of the data. On a continuing basis, 
efforts must also be devoted to evaluation and review of the 
center's methodology and products as one step in exploring 
and developing future methodology. 

Such a diverse menu of needs calls for survey and math- 
ematical statisticians or others with similar expertise, for 
staff trained in the subject of education and education policy 
and with analytic skills, and for staff experienced in the 
needs and practices of states, local education agencies, and 
other data providers. Competent leadership must set realistic 
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boundaries of what can be accomplished to protect the 
organization from unrealistic demands. Once having achieved 
and demonstrated an acceptable level of competence and 
accomplishment, an organization can expand and stretch as 
new needs are encountered. As needed, it also can seek 
short- or long-term outside assistance. In either case, how- 
ever, it is essential that what is to be done be done well and 
within the capabilities, knowledge, and experience of avail- 
able staff resources. 

The panel wishes to emphasize most forcefully its belief 
that the center staff itself must include sufficient expertise 
and the required leadership capability to understand, address, 
and respond to both the technical and related requirements 
of the demands placed upon it if it is to be perceived as 
(and in fact is to be) a competent statistical organization. 
Although it may require outside assistance to meet all of its 
commitments, it cannot, should not, and must not turn to 
others for its basic needs. 

Given the long history of education statistics and the 
almost universal agreement on their importance, our exposi- 
tion of necessary skills, related requirements, and essential 
attributes for a responsible statistical organization could be 
viewed as both naive and belaboring the obvious. Unfortu- 
nately, such is not the case--the situation at the center 
leaves much to be desired. During a period of rising 
demands for more and better information, staffing and bud- 
gets have declined significantly. Between fiscal years 1980 
and 1984, the number of full-time equivalent employees in the 
center decreased from 173 to 134, a 23 percent decrease 
(Congressional Research Service, 1985) and the number in 
early 1986 has dropped to 111. Adding insult to injury, a 
report issued in 1984 by the Committee on Government Oper- 
ations noted that during this same period of overall decline 
"the Department has required the Agency to fill 14 positions 
. . . with assigned staff because of RIFs (reductions in force) 
and reorganizations (elsewhere) within the Department" (Con- 
gressional Research Service, 1984:172). 

That the problem of adequate and competent staff is not 
of recent vintage is illustrated by a rather succinct quote 
from the report of the 1963 advisory group (U.S. Department 
of Health, Education, and Welfare, 1963:14): "The Office of 
Education [the predecessor organization to the center] does 
not have adequate professional staffing. ..." Of somewhat 
more recent vintage, it is instructive to ntite that in a mem- 
orandum to division directors in mid-1985* the director noted 
that efforts to improve quality at the center will not succeed 
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unless existing staff attitudes that "it doesn't make much 
difference what I do" are countered and overcome. 

During its study, the panel heard not only of the scarcity 
of staff, but also of the scarcity of skilK Serious questions 
were raised as to technical competence, particularly in the 
areas of sampling and survey methodology and of evaluation 
of the adequacy of products. As an example of this concern, 
of a staff of 128 permanent full-time members in March 1985, 
only 11 were classified as mathematical statisticians, and 
none was located in the office of the director. The panel 
was also presented with a number of examples of inadequate 
or incorrect presentation and interpretation of data by center 
staff. Furthermore, the center was unable to document any 
recent instances in which staff had prepared and presented 
technical papers before, or had participated in, meetings of 
professional associations, such as the American Educational 
Research Association, the American Statistical Association, 
the American Sociological Association, or the Population 
Association of America. 

We note the recent establishment by the American Sta- 
tistical Association of a research fellowship and associateship 
program, in cooperation with the center, which stands as a 
very positive development for the center. The program, sup- 
ported by a grant from the National Science Foundation, will 
bring to the center for periods of a year or so senior 
researchers and advanced graduate students to pursue 
research on any area related to the center's data or method- 
ology. 

The inability or unwillingness to respond to the existence 
of vacancies, particularly when the agency employment level 
is consistently below its ceiling and when its program 
obviously is in a state of growth, has been demoralizing and 
destructive. For example, an interval of about nine months 
to fill a mid-level statistical position (GS-11) raises doubts as 
to the commitment of the department to the objectives and 
needs of the center. Similarly, a personnel ceiling of 129 
compared with an on-board staff of 111 (of whom some 90 
are classified as professionals) at the time that the Congress 
has before it a request for a 50 percent budget increase 
(spring 1986) and that work has already begun on a new, 
major reimbursable project for another office in the depart- 
ment must also provide some pause. 

The recent reports prepared for the Committee on 
Government Operations of the House of Representatives 
(Congressional Research Service, 1984, 1985) also document a 
decrease of 28 percent in budget (when adjusted for infla- 
tion) over the five-year period beginning 1980, which resulted 
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in sharp curtailments in the ability of the aiaff to visit 
states to coordinate efforts, review activities, or otherwise 
interact with providers and users of its data. The effect of 
the budget and staffing curtailments also has had a direct 
negative effect on programs. The center's publications 
shifted away from more detailed analytic report to more lim- 
ited bulletin or statistical summaries, and even more discour- 
aging has been the absence in recent years of any efforts in 
validating any of the center's surveys or programs. 

These examples inevitably raise the question of leadership. 
The absence of leadership was mentioned by many who 
shared their views with the panel. Therefore, it seems 
obvious that a first requirement of any attempt to deal with 
the center's problems is for the leaders of the center to 
demonstrate their capabilities and regain the respect and sup- 
port, as well as control, of the staff. It also is essential 
that the lines of responsibility and authority throughout the 
center be clearly delineated, followed, and supported, and 
that leaders be held accountable and responsible for their 
actions or lack thereof. 

Recent changes in leadership, both at the center (in late 
1984) and in the department (in early 1985) appear to be 
resulting in some positive changes both in attitudes toward 
the center and in increased resources. The new leaders 
appear particularly realistic in their outlook: in the words of 
the present Assistant Secretary, Office of Educational 
Research and Improvement, thus far, the center "is not doing 
enough and not doing it well enough.** Both the assistant 
secretary and the director recognize that a large task lies 
ahead to "repair errors of the past and nurture for the 
future;** they have articulated three goals for the center: 

to repair the data base; 

to add qualitative information on the quantitative base; 
and 

to ensure that the center's products have both utility and 
quality. 

The work that is done by the statistical agency chiefly 
responsible for understanding the status of the U.S. educa- 
tional system requires stable and competent leadership and 
staff that must have substantial quantitative and subject- 
matter expertise. 
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o The panel recommends that the center identify the 
professional and technical staff required to successfully 
accomplish its mission and initiate an independent 
review and assessment of the technical and subject- 
matter qualifications of the present staff. 

o The panel recommends that the center initiate an 
active and continuing recruiting program to obtain 
staff with the needed skills that are lacking among the 
present staff. 

o The panel recommends that the center devote resources 
for the purposes of developing innovative approaches, 
including training and work assignments, to ensure the 
continuing technical growth and competence of the 
staff, and that staff accomplishment be rewarded by 
opportunity for further professional development. 

o The panel recommends that the lines of responsibility 
and authority throughout the center be clearly delin- 



o The panel recommends that the center, as needed, 
actively solicit the assistance and cooperation of its 
sister federal statistical agencies in addressing tech- 
nical issues or problems, to obtain short-run staffing 
assistance, or in carrying out various facets of its pro- 
gram. 

Given the extreme importance the panel places on the 
development of competent staff— and, in fact, in carrying out 
its other recommendations for improvements in the center's 
program— the panel fully expects that the center will curtail 
its program as may be necessary to free the resources neces- 
sary to improve its current operations. 

To institutionalize the central place of quality in the work 
of the center, it is vital that the issue of statistical quality 
explicitly be made a central component of the center, with a 
focal point in an Office of Statistical Standards, as recom- 
mended above by the panel. 

In addition to technical competence, many factors enter 
into the perception and reality of an organization recognized 
as outstanding, one responsive to demands and innovative in 
meeting needs. We have already highlighted the importance 
of leadership; without vision, direction, and commitment, no 
organization can long sustain excellence. At a somewhat 
lower level-but vital nonetheless-are a host of activities or 
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requirements that set the tone, sustain the momentum, and 
provide the basis for continual growth and improvement. 
These include staff recruitment, training, and development; 
the conscious fostering of communication within the organi- 
zation and between the organization and its users and pro- 
viders; coordination of efforts among all the participants; 
exposure of the organization to outside review and critique 
of its procedures, methodologies, and products; efforts at 
consultation, both to provide others with the benefits of the 
center's expertise and to obtain from others the fruits of 
their wisdom and experience; and finally, the product of all 
the above, a sense of morale, that elusive element that binds 
an organization and makes possible continued accomplishment. 

o The panel recommends that the center staff be given 
both responsibility and authority for the conduct of 
individual projects and, accordingly, be held account- 
able for meeting the established standards of 
acceptable performance, including timeliness. 

o The panel recommends that the center recognize the 
value and importance of participating in meetings and 
activities of professional and technical organizations, 
including the preparation, presentation, and publication 
of papers and articles describing aspects of the work 
of the center and its problems, and that the center 
devote, and if necessary divert, resources to ensure 
and support staff participation. 



CONTRACTING OUT--OR NOT 

Federal government agencies— statistical and nonstatistical— 
have a range of options in arranging for the performance of 
their statistical activities. The first level of choice is 
whether to perform the project with in-house staff or to use 
the services of an outside organization. If an outside organi- 
zation, choices include another federal statistical agency, a 
private survey firm, a university survey center, a nonprofit 
research institute, a public-interest association, or a unit of 
state or local government. 

Major federal statistical agencies exhibit many different 
configurations in the arrangements by which they perform 
their mainstream activities. The Bureau of the Census, for 
example, serves as a survey data collection and processing 
agent for many federal statistical and nonstatistical agencies. 
In contrast, the Center for Statistics has relied heavily on 
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statoi and local education agencies for the primary collection 
of elementary and secondary school data and has contracted 
out most of its other data collection activities, including such 
major longitudinal surveys such as High School and Beyond. 
In addition, it has contracted out for some processing of data 
received from state agencies and from institutions of higher 
education. Over all, very little of its work has been carried 
out directly by its own staff. 

The question of the proper role for outside contract ser- 
vices in the performance of a statistical center's primary or 
core activities is a very difficult one. In the case of the 
center, a basic problem is that a professional staff the size 
of the center's (about 90 in spring 1986) cannot be expected 
to develop and maintain an appropriate level of expertise 
across all of the technical and subject-matter fields that are 
required in carrying out its mission and program. 

The general adequacy and quality of the survey work and 
other collection and processing activities carried out by the 
center depend heavily on its operational capacity to plan, 
manage, and evaluate technical projects. Similarly, the 
ability to carry out supervisory or overview functions to the 
maximum benefit of the center depends heavily on the exper- 
ience of the staff in having participated directly in one or 
another of the many facets of direct data collection and pro- 
cessing. Only with such experience can one be expected to 
recognize fully, understand the meaning of, and react to the 
measures of quality control, the implications of actions taken 
or deferred, the significance of patterns of expenditures, and 
the proper choices among alternatives when plan and perfor- 
mance differ. 

o The panel recommends that the center undertake to 
conduct some projects directly in-house, in order to 
provide the opportunity for staff to become exposed to 
and experienced in the variety and multitude of tech- 
nical activities, problems, and difficulties inherent in 
such an undertaking and to provide technical compe- 
tence for the staff in subsequently supervising outside 
contractors. 

With regard to contracting with private organizations, the 
center has had a range of experiences. The longitudinal 
studies have received generally favorable reviews from 
knowledgeable observers. In contrast, the center has had 
some unpleasant experiences in the nonperformance or inade- 
quate performance of support contractors in the area of 
operational processing and checking of data from educational 
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institutions. Although it is not possible to guarantee impec- 
cable, continuing performance on the part of any contractor, 
it is a major responsibility of center management to recog- 
nize early warnings of problem areas and to take immediate 
and decisive action to correct them. From the opposite 
viewpoint, contractors have indicated their share of difficul- 
ties in dealings with the center. Qualified technical staff to 
evaluate and review the contractor's work have been and 
continue to be in short supply, with consequent delays to the 
program. Specifications once agreed upoir have been changed 
without notice or much discussion, needed decisions have 
been delayed, the center has spoken often with many differ- 
ent voices— all adding to confusion, loss of time, and less 
than efficient performance. In each instance, the availability 
of knowledgeable, competent, and experienced staff is an 
essential element, first, in anticipating and preventing prob- 
lems and, second, in dealing with them expeditiously and 
efficiently if and when they occur. As noted earlier, ensur- 
ing data quality must be the center's responsibility. 

The panel believes it would be very much to the center's 
advantage to develop close relationships with other major 
federal statistical agencies, in particular, the Bureau of the 
Census. Given the bureau's special role in carrying out work 
for other federal agencies; given its mix of resources of staff 
experienced in all aspects of data collection and processing, 
including the use of both sample surveys and administrative 
records; and given its past experience in undertaking work 
for the center, the ^?nei urges the center to explore the 
establishment of ?. continuing, substantial relationship 
between itself and the bureau. Such a relationship might 
include, among other activities, the conduct of at least one 
major program for the center by the bureau. Such an under- 
taking, if feasible, would permit staff of the center to be 
exposed regularly to technical staff of the bureau and to the 
problems inherent in all phases of data collection; to partici- 
pate in the determination of all aspects of the undertaking, 
including dealing with problems and developing solutions and 
observing data collection u nd processing; and to call on the 
bureau's staff for technical and other assistance in areas 
other than those involved in their joint relationship. The 
period of outstanding cooperation a decade ago among staff 
of the Department of Health, Education, and Welfare, of the 
center, and of the Census Bureau during the development and 
conduct of the Survey of Income and Education can serve as 
a model. 
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o The panel recommends that the center establish a con- 
tinuing relationship with the Bureau of the Census and 
explore the feasibility of joint participation in one or 
more major programs. 



The data products of the Center for Statistics are the cul- 
mination of the center's effort, and therefore are the pri- 
mary vehicle through which the work of the center should be 
judged. Ineffective publications and incomplete or incorrect 
data tapes and documentation that fail to serve the needs of 
data users, publications that contain outdated data, reports in 
which text and data are inconsistent, reports that fail to 
detail methodology and provide error measures-all these have 
been ascribed to the center. Unfortunately, our review of 
examples of the center's publications confirms the view that 
the center has had more than its share of problems. In fact, 
the initial support within the department for a study of the 
center resulted in large part from concerns with the presen- 
tation of results in the center's publications. Many of the 
recommendations already proposed in the report, if implemen- 
ted, would result in improvements in th3 center's publica- 
tions. 

The critical lacks in many of the center's publications 
come quickly to the fore when its publications are compared 
with those of other federal statistical agencies: timeliness; 
the absence of detail on the definitions used or on the 
sources of the data themselves; any sense of how the data 
we/e treated to get them into the form in which they were 
presented; the treatment of problematic data elements; esti- 
mates of the sampling or nonsampling error present in the 
numbers; disclosure avoidance techniques; information on the 
use of imputation or weighting procedures. Users find them- 
selves in the dark, unable tG make appropriate and necessary 
judgments on quality and left with a feeling that the datr 
are from an unknown source, of unknown quality, and have 
been subject to unknown editing and imputation procedures. 
As a consequence, their ability to make maximum use of the 
data is restricted by their necessary caution. Such deficien- 
cies lead the panel to emphasize the need for the center to 
establish standards and to developing mechanisms for moni- 
toring their implementation. Of course not all center publi- 
cations exhibit the problems we have identified; their exis- 
tence merely heightens the contrast with those publications 
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and reports that fail in their objective to inform and 
enlighten the user. 

Our review also indicates clearly the need to upgrade the 
technological base for producing publications. Few if any of 
the present publications demonstrate concern with readability, 
clarity in presentation, and the use of graphs or charts to 
aid users. The center-and its publications— would benefit 
through the application of graphic tools for display and ana- 
lysis. Similar attention should be directed to dissemination 
and distribution of publications. 

An additional area of concern regarding publications and 
the dissemination of information arose as a result of the 
recent reorganization of the center, which moved much, if 
not all, of the dissemination function cut of the center. 
Given that a major function and responsibility of a statistical 
organization is to understand how and by whom its data are 
being used, the panel believes that the organization must 
interact directly and continually with the users of its prod- 
ucts. Direct interaction also serves a number of other very 
important functions, such as allowing those who best under- 
stand the data to explain its limitations and meaning to 
potential users; providing users with the opportunity of 
learning about other types of data that might better serve 
their needs or provide a broader understanding of issues 
being studied; and, finally, providing producers with the 
opportunity of determining future data needs. The important 
nature of this activity leads the panel to conclude that close 
attention must be paid to the operation of the information 
function under the new organizational arrangement. 

o The panel recommends that the center institute a pub- 
lications policy that sets forth the types of publica- 
tions—analytic reports, news releases, descriptive 
reports, statistical summaries and digests, methodolog- 
ical reports, etc.— and other forms of information, such 
as data tapes, diskettes, on-line reporting capabilities 
to be made available; audiences to be served; and the 
frequency of release of data. 

o The panel recommends that the center establish, pub- 
lish, and adhere to a set of fixed release dates for 
selected key education statistics. 

o The panel recommends that the publications policy 
require that center reports contain information on def- 
initions used; a description of the data collection 
methodology; provide a measure of reliability, if 
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applicable; discuss other posfiluc purees of error; and 
detail other sources of relevant or similar data. Com- 
pendia of statistics, such as the Condition of Educa- 
tion, should, at a minimum, follow the example of the 
Statistical Abstract of the United States in providing 
references to source information; similarly, advance or 
brief highlights reports should contain appropriate 
references to the sources of such information. 

The panel recommends that the Office of Educational 
Research and Improvement and the Center for Statis- 
tics, given the current organizational separation of the 
dissemination of data from the preparation phase, mon- 
itor and review closely the release of information 
through the new arrangement and the quality and 
extent of interaction with users to ensure that 
appropriate guidance on sources, uses, interpretations, 
and limitations of the data is provided to those receiv- 
ing the information. 

The panel recommends that the center initiate a 
comprehensive review of its technological capabilities, 
with the objective of developing a short-range plan for 
providing state-of-the-art capability in the dissemina- 
tion and distribution of its products. 




4 



Recommendations 



This chapter brings together all the panel's recommendations 
for the Center for Statistics. They are presented in the 
order in which they appear in the report. 

The panel believes that its recommendations provide a 
realistic approach to addressing the issues and problems that 
face the center, both from the realities of the past and 
present, and from the longer-range perspective of the center 
as we envision it in the years ahead. Given our findings, we 
believe that what we propose is critical to the survival of 
the center as a viable and credible organization. Our recom- 
mendations are a comprehensive and integrated set, a package 
that, taken as a whole, can make the center into a credible 
statistical agency. 

In setting forth its recommendations, tbe panel is fully 
mindful of the current budget situation. First, many of its 
recommendations can be implemented within the current bud- 
getary capacity of the center and will result in significant 
improvements in the quality, timing, and accuracy of what it 
does and how it does it. Second, the center must be pre- 
pared as needed to curtail program, scope, or activity in 
order to improve the quality of its work and products. In 
the view of the panel, less, done well, is far superior to 
more, done poorly. The panel recognizes and accepts that 
over time the center will require additional funding; however* 
it believes that such increases must be judged against the 
progress of the center in meeting the standards expected of 
it and should reflect an expected expansion in its program as 
it demonstrates its progress. 

Finally, the panel wishes to emphasize the seriousness of 
its view that there can be no defense for allowing the center 
to continue as it has for all too long. It is past time for 
those in positions of responsibility to face up to the risks 
and dangers of perpetuating the myriad and continuing prob- 

54 



64 



55 



lems of the center. Without strong and continuous commit- 
ment and demonstrated determination to undertake wide-rang- 
ing actions to change both the image and reality of the cen- 
ter, we are unanimous in our conviction that serious consid- 
eration should be given to the more drastic alternatives of 
abolishing the center and finding other means to obtain and 
disseminate education data. For our part, we continue to 
believe strongly that the center has a future and that the 
preferred course of action is to begin the process of 
improvement. 



MISSION, ROLE, AND RESPONSIBILITIES 

The panel recommends that the mission of the Center for 
Statistics, as stated in Section 406(b) of the General 
Education Provisions Act, as amended (20 U.S.C. i22le-l), be 
strengthened to clearly establish and define the role of the 
center in assisting the Secretary of Education in determining 
the data needs for assessing the condition of education in 
the United States, and for accepting responsibility and 
accountability for ensuring the availability of the necessary 
data. 



The panel recommend 
for the center ar^ 
through itw requec'i 
requests for direct 
through its calls for t 



Vngress demonstrate its support 
n through its budget actions, 
'<d statistical advice, through 

- from center officials, and 
1 



The panel recouime;, aat the Sectary of Education 
demonstrate strong and continued sup^rt for the center and 
its mission through depart .^ent budget requests, public state- 
ments, and in appearances before Congress. 

The panel recommends *hat Congress arid the secretary rec- 
ognize and state their support both for the nonpartisan 
nature of the center and its statistical independence. 

The panel recommends that the secretary designate the cen- 
ter as the functional agency responsible for coordination and 
technical review of all data collection within the department. 

The panel recommends that the secretary designate the cen- 
ter as the focal point of releasing statistical information on 
education, 
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The panel recommends that the center assume accountability 
for the process that leads to the determination of the con- 
tent of the data to be collected. 



CONTENT AND DIRECTION 
OF DATA COLLECTION PROGRAMS 

The pane! recommends that the leadership of the center 
develop mechanisms to assist policy makers in determining 
their data needs and that the center continue to involve its 
users in dialogue to determine the most relevant end appro- 
priate content of its data collection efforts. 

The panel recommends that the center continue the compila- 
tion of program, staff, and financial data from the states, 
but that it undertake an analysis, jointly with representatives 
from the states, to ensure that the present program meets 
both center and state requirements for usefulness, relevance, 
quality, and reliability. In addition, the content of the data 
on the condition of education, originating with the states, 
should be be monitored periodically with the intent of 
improving the content and thus its joint usefulness to the 
center and the states. 

The panel recommends that the center initiate a sample-based 
program of data collection focused on individual classrooms 
and students, designed to facilitate better understanding of 
the relationships between educational inputs, processes, and 
outcomes. The sample would not necessarily be able to pro- 
vide adequate estimates for all individual states, although it 
should be organized so that states could choose to augment 
the sample to provide state data. 

The panel recommends that sample-based data on educational 
inputs, processes, and outcomes be made readily available, on 
a timely basis, to all interested users in the form of public- 
use tapes, with appropriate masking of characteristics to pre- 
clude the identification of individual schools, teachers, or 
students. The panel believes that the usefulness of center 
data would be greatly enhanced if its data become widely 
available as a resource for both policy and research use. 

The panel recommends that the. center continue to explore 
the inclusion of longitudinal features in its sample-based sur- 
vey data and that, as a minimum, small-scale longitudinal 
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studies be mounted regularly as part of the center's data 
collection efforts. 



ESTABLISHING A FRAME OF REFERENCE 

The panel recommends that the center develop a conceptual 
framework for organizing its program and for setting priori- 
ties in light of available resources. 



The panel recommends that Congress modify the mandate of 
the Advisory Council on Education Statistics to a role of 
technical review and oversight, rather than responsibility for 
the establishing of statistical standards. 

The panel recommends that the Secretary of Education, in 
support of the center, provide timely written response to the 
concerns raised by ACES in its annual reports to the secre- 
tary, and that Congress periodically hold hearings at which 
both the secretary and representatives of ACES provide tes- 
timony on the status, problems, and progress of the center 
and the concerns and advice of ACES. 

The panel recommends that the members of ACES be selected 
for their expertise and competence in areas directly related 
to the center's program. 

The panel recommends that the center establish one or more 
continuing technical advisory groups, in addition to ACES, to 
meet at least twice per year to review, comment, and advise 
on the methodology used in the conduct of its work, and to 
provide the director with objective, outside, independent 
review 0 f the technical capability of the center's staff or of 
its products. As required by its changing program needs, the 
center also should establish ad hoc advisory groups repre- 
senting the unique technical and subject area skills. 
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IMPROVING DATA QUALITY 

Statistical Standards 

The panel recommends that the center develop, publish, dis- 
seminate, and implement standards to guide the conduct of 
all phases of its work, from development of objectives 
through collection, follow-up, and processing and including 
the preparation, review, analysis, and publication of results. 

The panel recommends that the center establish an Off^e of 
Statistical Standards and Methods and move expeditiously to 
recruit and appoint a chief statistician at the leve! of assis- 
tant director, with responsibility for the establishment and 
maintenance of statistical standards throughout the center. 

The panel recommends that the center, in concert with 
appropriate state and local education agencies (LEAs) and 
representatives of institutions of higher education, institute 
the development and publication of uniform definitions and 
their continuing review, to ensure the collection of consis- 
tent data, such as between different LEAs and states, Such 
information should be made available regularly in reference 
handbooks. 

The panel recommends that the center undertake a continuing 
program of evaluation of its methodology and its programs. 

The panel recommends that the center disseminate widely the 
results of its methodological research. Publications in pro- 
fessional journals and presentations at conferences should be 
strongly encouraged. 

The panel recommends that the center require that its 
reports contain information on definitions used in studies and 
a description of the data collection methodology, provide a 
measure of reliability, if applicable, and discuss possible 
sources of error. 



Redesign of the Elementary/Secondary Data Programs: 
An Example 

The panel recommends that the center continue to explore 
most carefully the appropriate balance between the continuing 
use of state and local administrative record sources and the 
use of sample surveys. Although supportive of the use of 
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sample surveys, the panel does not brieve that the center 
has fully resolved the issues involved in shifting the elemen- 
tary/secondary data collection system from one based on 
administrative records to one based solely on an integrated 
sampling survey approach. The panel endorses the efforts of 
the center in seeking new approaches to accomplishing its 
mission and suggests that small-scale testing within one or a 
few states be undertaken to establish procedures, define 
problems and develop solutions, refine the approach and, 
overall, assess the feasibility and cost of the survey proposal. 



Detecting and Reducing Error 

The panel recommends that the crenter routinely make its 
users aware of the quality of the data and the limitations 
inherent in their use. 

The panel recommends that the center foster a climate of 
openness in dealing with all elements of its constituency, 
including other parts of the department, other government 
agencies, Congress, providers and users of its data, and its 
professional peer groups. 

The panel recommends that the center initiate a comprehen- 
sive program to assess and improve, where necessary, the 
quality, consistency, and reliability of data obtained from 
state and local agencies, from institutions of higher educa- 
tion, and from other sources. Meetings, discussions, visits, 
and technical assistance, including the training of local and 
state officials on what is required of them and their systems, 
must comprise a major component of the program. 

The panel recommends that the center establish stancf *rds for 
review of data releases and institute procedures to effect 
them. 

TIMELINESS 

The panel recommends that the center explore and develop 
approaches for expediting response from participants in its 
studies and techniques for producing usable advance results; 
use management information systems for monitoring progress 
through all phases of data collection and reporting; and 
establish procedures Tor taking prompt and appropriate 
actions when required. Similar techniques and monitoring 
controls should be required as integral components of work 
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done by contractors to the center and should be followed 
closely to ensure prompt adherence to schedule, with penal- 
ties for failure to accomplish work on time. 



RESOURCES—STAFF AND BUDGET 

The panel recommends that the center identify the profes- 
sional and technical staff required to successfully accomplish 
its mission and initiate an independent review and assessment 
of the technical and subject-matter qualifications of the 
present staff. 

The panel recommends that the center initiate an active and 
continuing recruiting program to obtain staff with the needed 
skills that are lacking airing the present staff. 

The panel recommends that the center devote resources for 
the purposes of developing innovative approaches, including 
training and work assignments, to cfisuire the continuing 
technical growth and competence of the staff, and that 
accomplishment be rewarded by opportunity for further pro- 
fessional development. 

The mntl recommends that the lines of 4 ;sponsibility and 
authority throughout the center be clearly delineated. 

The -^ncl recommends that the center, as needed, actively 
solicit the assistance and cooperation of its sister federal 
statistical agencies in addressing technical issues or problems, 
to- obtain short-run staffing assistance, or in carrying out 
various facets of its program. 

The panel recommends that the center staff be given both 
responsibility and authority for the conduct of individual 
projects and, accordingly, be held accountable for meeting 
the established standards of acceptable performance, including 
timeliness. 

The panel recommends that the center recognize the value 
and importance of participating in meetings and activities of 
professional and technical organizations, including the pre- 
paration, presentation, and publication of papers and articles 
c '• vibing aspects of the work of the center and its prob- 
lem, and that the center devote, and if necessary divert, 
resources to ensure and support staff participation. 
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CONTRACTING OUT--OR NOT 



The panel recommends that the center undertake to conduct 
some projects directly in-house, in order to provide the 
opportunity for staff to become exposed to and experienced 
in the variety and multitude of technical activities, problems, 
and difficulties inherent in such an undertaking and to pro- 
vide technical competence for the staff in subsequently 
supervising outside contractors. 

The panel recommends that the center establish a continuing 
relationship with the Bureau of the Census and explore the 
feasibility of joint participation in one or more major pro- 
grams. 



The panel recommends that the center institute a "'iblications 
policy that sets forth the types of publications-analytic 
reports, news releases, descr e reports, statistical sum- 
maries and digests, methodological reports, etc.-and other 
forms of information, such as data tapes, diskettes, on-line 
reporting capabilities to be made available; audiences to be 
served; and the frequency of release of data. 

The panel recommends that the center establish, publish, and 
adhere to a set of fixed release dates for selected key edu- 
cation statistics. 

The panel recommends that the publications policy require 
that center reports contain information on definitions used; a 
description of the data collection methodology; provide a 
measure of reliability, if applicable; discuss other possible 
sources of error; and detail other sources of relevant or 
similar data. Compendia of statistics, such as the Condition 
of Education, should, at a minimum, follow the example of 
the Statistical Abstract of the United States in providing 
references to source information; similarly, advance or brief 
highlights reports should contain appropriate references to 
the sources of such information. 

The panel recommends that the Office of Educational 
Research and Improvement and the center, given the current 
organizational separation of the dissemination o r data from 
the preparation phase, monitor and review clos&'v the release 
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of information through the new arrangement and the quality 
and extent of interaction with users to ensure that appropri- 
ate guidance on sources, uses, interpretations, and limitations 
of the data is provided to those receiving the information. 

The panel recommends that the center initiate a comprehen- 
sive review of its technological capabilities, with the objec- 
tive of developing a short-range plan for providing state-of- 
the-art capability in the dissemination and distribution of its 
products. 
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Appendix A 
Organizational Structure of the Center 



During the panel's tenure, the Center underwent a number of 
reorganizations, including a renaming, from the National Cen- 
ter for Education Statistics to the Center for Statistics. 
Shown on the following pages are the organizational struc- 
ture of the center, as of April 1986, and a listing of the 
program content and staffing of each of the organizational 
components. 
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CONDITION OF EDUCATION DIVISION 



Condition of 
Education 
Division 

20 professional 
3 support 

Indicators 

Condition of Education 
Digest of Education Statistics 
Projections of Education Statistics 
Allocations 

Data on Vocational Education 



ELEMENTARY/SECONDARY EDUCATION 
STATISTICS DIVISION 



Elem/Sec 
Education 
Statistics 
Division* 

1 professional 

1 support 



Elementary-Secondary Redesign 



General Surveys 
and 

Analysis Bran ch j 
11 professions 
1 support 



Common Core of Data (CCD) 

CCSSO Education Data Improvement Project 



Special Surveys 
and 

Analysis Branch 



9 professional 
1 support 



Public/Private School (including libraries) Surveys 

School and Staffing Surveys 

LEA Teacher Supply and Demand Survey 



so 
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'OSTSECONDARY EDUCATION STATISTICS DIVISION 



Postsecondary 
Education 
Statistics 
Division 

2 professional 

1 support 



General Surveys 
and 

Analysis Branch 



IS professional 
2 support 



Higher Education General Information System 
(HEGIS) 

Integrated Postsecondary Education Data System 
(IPEDS) 



Special Surveys 
and 

Analysis Branch 



8 professional 
2 support 



National Postsecondary Student Aid Study (NPSAS) 

Recent College Graduates (including transcript) Surveys 

State Higher Education Ex-scutive Officers (SHEEO)/CS Network 



EDUCATION OUTCOMES DIVISION 



Education 
Outcomes 
Division 

2 professional 
1 support 



Longitudinal 
Studies Branch 



8 professional 
I support 



National Education Longitudinal Study of 1988 (NELS 88) 

High School and Beyond 

National Longitudinal Survey (NLS-72) 



Cross Sectional 

and SpecizJ 
Studies Branch 



9 professional 
2 support 



National Assessment of Educational Progress (NAEP) 

International Studies 

Fast Response Survey System (FRSS) 



81 



Appendix B 
Biographical Sketches 
of Pane*? Members and Staff 
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and planning at General Motors Corporation, Detroit, Michi- 
gan. He was formerly director of the U. S. Bureau of the 
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utilization. He is a fellow and former vice president of the 
American Statistical Association, a member of the Interna- 
tional Statistical Institute, and on the board of directors of 
the Social Science Research Council. He received undergrad- 
uate degrees in advertising and marketing from Woodbury 
Business College and California State University and an 
M.B.A. degree in marketing from the University of California, 
Los Angeles. 

ANTHONY S. BRYK is associate professor in the Department 
of Education at the University of Chicago, chairs the depart- 
mental computer committee, and oversees the operation of 
the department's computer center. He has been both admin- 
istrator and senior research associate at the Huron Institute, 
a firm of applied research, evaluation, and policy analysis. 
His current research interests are in applications of hierarch- 
ical linear models to educational research, the sociopolitical 
organization of the evaluation context and its implications 
for the role and training of applied social scientists, and the 
social organization of Catholic schools. He received a B.S. 
degree from Boston College in chemistry and an M.Ed. degree 
from Harvard University in measurement and statistics. 

MICHAEL L. COHEN is a faculty research associate in the 
School of Public Affairs at the University of Maryland. He 
is a former research associate to the Com -ittee on National 
Statistics and served as consultant to the Panei to Evaluate 
the National Center for Education Statistics. His primary 
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interests are in data analysis, regression, and sample design 
and estimation. He received a B.S. degree from the Univer- 
sity of Michigan and MA. and Ph.D. degrees in statistics 
from Stanford University. 

ROBERTO M FERNANDEZ is an assistant professor in the 
Department of Sociology at the University of Arizona and a 
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on schooling and early labor force activities of Hispanic 
youth, using data from the the center's longitudinal study, 
High School and Beyond, and the Department of Labor's 
national longitudinal study of youth labor market experience. 
His work also includes the institutional factors that deter- 
mine the structure of labor markets, the role of social net- 
works in recruitment to social movements, and the role of 
interorganizational networks in policy domains. He recently 
served on the National Research Council's committee on 
youth employment. He received a B.A degree from Harvard 
University and MA. and Ph.D. degrees in sociology from the 
University of Chicago. 

CHRISTOPHER JENCKS is professor of sociology and urban 
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the development of higher education in the United States, 
the role of education in social stratification, and standardized 
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English from Harvard College. 

F. THOMAS JUSTER is director of the Institute for Social 
Research and professor of economics at the University of 
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National Statistics, chair of its Panel on Statistics of Supp'v 
and Demand for Precollege Science and MathcmatL, 
Teachers, and chair of the American Economic Association 
Committee on the Quality of Economic Statistics. He is a 
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B.S. degree from Rutgers University A n4 a Ph.D. degree in 
economics from Columbia University. 
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